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Despite  a  variety  o-f  regulations  and  directives  on  the 
subject  o-f  acquisition  data,  there  are  currently  no  pub¬ 
lished  criteria  which  will  aid  Program  Managers  in  their 
decision  to  purchase  acquisition  data  for  a  particular  i tee. 
This  research  effort  proposes  a  heuristic  decision-making 
model  which  will  aid  Program  Managers  or  Data  Managers  in 
their  acquisition  data  purchasing  decisions. 

The  researchers  first  identified,  through  a  literature 
search  and  a  series  of  personal  interviews,  eleven  criteria 
relevant  to  the  acquisition  data  purchasing  decision.  These 
criteria  were  then  ranked  and  weighted  by  acquisition  data 
experts  using  a  second  set  of  personal  interviews. 

By  rating  items  for  which  acquisition  data  was  pur¬ 
chased  and  later  used  in  a  competitive  reprocurement,  and 
items  for  which  no  acquisition  data  was  purchased,  the 
researchers  developed  two  decision  points  for  their  model. 
These  two  points  dictate  whether  or  not  the  model's  user 
should  purchase  acquisition  data  for  the  item  being  rated. 

The  researchers  tested  the  model  by  randomly  selecting 
items  for  which  a  decision  has  already  been  made  to  purchase 
or  not  purchase  the  acquisition  data.  Each  item  was  rated 
using  the  decision-making  model  and  the  model’s  recommended 
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decisions  were  compared  to  the  actual  decisions  made.  This 
test  procedure  resulted  in  a  77. 8X  success  rate  where  the 
model  accurately  predicted  the  actual  decision  made  by  the 
Program  Manager. 

The  researchers  recommended  that  further  tests  be 
performed  to  determine  the  ability  of  the  model  to  predict 
various  combinations  of  acquisition  conditions,  program 
types  and  purchase  item  types. 


A  HEURISTIC  APPROACH  TO  DECISION-MAKING 


FOR  THE  PURCHASE  OF  ACQUISITION  DATA 


I •  Introduction 


Background 

During  the  1982-1983  time  period  the  rapid  increase  in 
costs  associated  with  the  acquisition  of  spare  parts  for 
government  weapon  systems  (an  $837.5  million  increase  during 
Fiscal  Y ear  1982)  has  been  highlighted  to  the  Congress,  the 
general  public,  the  Department  of  Defense  (DOD) ,  and  Air 
Force  Leadership  <2sl-l). 

In  response,  the  Air  Force  Management  Analysis  Group 
(AFMAG)  -  Spare  Parts  Acquisition  was  formed  at  the  direction 
of  Secretary  of  the  Air  Force,  Verne  Qrr,  and  Air  Force  Chief 
of  Staff,  General  Gabriel,  and  formally  chartered  by  the  Air 
Force  Vice  Chief  of  Staff,  General  Skantz,  on  20  May  1983. 

One  of  the  primary  purposes  of  AFMAG  was  to  determine  the 
reasons  for  the  increase  in  the  costs  of  spare  parts  for  Air 
Force  weapon  systems  <2:1-1). 

The  major  factor  behind  the  increased  costs  for  spare 
parts  cited  by  AFMAG  was  a  lack  of  competition  in  the  spare 
parts  acquisition  process.  Since  1973  the  percentage  of  spare 
parts  competitively  acquired  by  the  Air  Force  has  declined 
(See  Figure  1)  from  a  high  of  37. 551  to  20.7%  in  1982 
<2:2-12) . 
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Figure  1.  Air  Force  Spare  Parts  Competition 

According  to  AFMAG,  the  four  main  reasons  for  this  decreased 
competition  rate  Mere: 

1.  Fielding  of  New  Weapon  Systems:  Between  1976  and 
1982  new  weapon  systems  (A-10,  F-15,  F-16)  accounted  for  32% 
of  all  Air  Force  spare  part  requirements.  It  is  general ly 
infeasible  to  compete  for  spars  parts  early  in  the  produc¬ 
tion  process  due  to  numerous  on-going  engineering  design 
changes  <2:2-14). 

2.  Bureaucratic  Processes:  These  are  regulatory  and 
legal  constraints,  such  as  Environmental  Protection  Agency 
requirements,  that  increase  admini strati ve  burdens  and  lower 


the  pro-fit  margins  of  defense  contractors.  The  net  result 
has  been  a  reduction  of  the  defense  industrial  base  from 
6000  suppliers  in  1964  to  3500  in  1980.  As  the  base 
shrinks,  the  ability  to  achieve  competition  is  reduced 
(2:2-19) . 

3.  Proprietary  Rights:  When  a  contractor  asserts  that 
a  part,  component  or  process  was  developed  at  private  ex¬ 
pense,  all  engineering  data  describing  that  part  or  process 
is  delivered  to  the  Air  Force  with  restrictions  that  pre¬ 
clude  the  use  of  that  data  for  competitive  spares  purchasing 
(2:2-16) . 

4.  Inadequate/Non-Ex i stent  Acquisition  Data:  The  en¬ 
gineering  data  necessary  to  support  the  competitive  acquisi- 
tion  of  spares  is  created  as  part  of  the  design  and  initial 
production  of  the  weapon  system,  but  Air  Force  efforts  to 
obtain  this  data  have  not  been  effective.  Approximately  16X 
of  Air  Force  spare  parts  that  are  eligible  for  competitive 
procurement  are  instead  purchased  on  a  sole  source  basis 
because  the  requisite  acquisition  data  is  either  missing  or 
inadequate  (2:2-15). 

Of  these  four  factors,  AFMAG  described  the  inadequacy 
and  unavailability  of  acquisition  data  as  being  the  greatest 
inhibitor  towards  increasing  competition  in  the  procurement 
of  spare  parts  (2:3-3).  Clearly  the  procurement  of  accu¬ 
rate,  complete  acquisition  data  is  essential  in  reducing  the 
high  cost  of  Air  Force  spare  parts. 


Problem  Statement 

Although  there  are  a  variety  of  Directives,  Regula¬ 
tions,  Pamphlets,  etc.,  that  address  the  subject  o-f  acquisi¬ 
tion  data,  there  is  no  published  methodology  that  helps  Air 
Force  Program  Managers  determine  which  parts  under  their 
responsibility  will  require  acquisition  data  and  which  parts 
will  not  require  acquisition  data. 

Definitions 

The  -following  terms  used  throughout  this  report  are 
de-fined  as  -follows: 

Engineering  Data.  Engineering  data  is  the  master 
baseline  documentation  -for  a  system.  It  describes  parts, 
assemblies,  and  equipments  and  the  way  they  go  together  to 
make  up  a  system  (3:54). 

Acquisition  Data.  The  acquisition  data  -for  a 
specific  hardware  item  includes  the  engineering  data  for 
that  item,  but  may  also  require  and  include  additional  data, 
such  as  item  peculiar  test  data  or  packaging  data,  that  is 
needed  to  acquire  the  part  competitively  (3:35). 

Justification 

The  Department  of  Defense  purchases  appr ox i matel y  six 
million  pieces  of  technical  data  each  year  at  a  cost  of 
nearly  $2  billion  (22:1).  In  1983  the  Air  Force  alone  ac¬ 
quired  approximately  750,000  pieces  of  data  which  were  ulti¬ 
mately  stored  at  the  Air  Force  Engineering  Data  Support  Cen¬ 
ter  (AFEDSC)  located  at  Wright-Patterson  AFB,  Ohio  (3-134). 


The  storage,  handling,  and  administration  of  this  data  at 
the  AFEDSC  costs  * 2.2  million  per  year  (3:136). 

When  considering  the  costs  involved  with  purchasing  and 
storing  data,  it  is  significant  to  note  that  some  estimates 
have  found  that  as  little  as  5  percent  of  the  acquisition 
data  bought  by  the  Air  Force  has  actually  been  used  in  the 
competitive  acquisition  of  spare  parts  (24:2). 

In  their  study,  AFMAG  found  that  the  AFEDSC  has  a  10% 
error  rate  in  processing  data  and  that  the  facility  was  able 
to  fill  only  40%  of  user  requests.  The  primary  reason  for 
this  was  that  the  Center’s  equipment  and  procedures  (both  in 
need  of  modernization)  are  unable  to  handle  the  large  volume 
of  data  (3: 134) .  The  AFMAG  also  noted  that  problems  asso¬ 
ciated  with  the  cost  estimation  of  data,  proprietary  data 
rights,  the  adequacy  of  data  received,  and  the  definition  of 
data  requirements  can  all  be  partially  attributed  to  the 
ratio  of  trained  manpower  to  the  sheer  volume  of  data  to  be 
reviewed  (2) . 

The  Air  Force  would  greatly  benefit  from  a  reduction  in 
the  volume  of  unnecessary  acquisition  data.  Data  purchas¬ 
ing,  handling  and  storing  costs  would  be  reduced.  A  reduced 
volume  of  acquisition  data  would  ease  manpower  shortages  in 
the  management  functions  associated  with  data  such  as  cost 
estimation  and  the  determination  of  data  adequacy.  Finally, 
a  reduction  in  data  volume  would  reduce  the  administrative 
burden  on  AFEDSC,  enabling  that  organization  to  reduce  their 
error  rate  and  provide  better  service  to  users. 


Research  Objectives 

1.  To  identify  and  rank  order  the  factors  relevant  in 
a  decision  to  procure  acquisition  data  for  use  in  the  com¬ 
petitive  acquisition  of  spare  parts. 

2.  To  use  the  identified  factors  in  building  a  model 
which  will  assist  Air  Force  Program  Managers  in  their  deci¬ 
sion  concerning  the  procurement  of  acquisition  data. 

3.  To  test  the  validity  of  the  constructed  model. 

Research  Questions 

1.  What  criteria  should  be  considered  in  determining 

whether  acquisition  data  will  be  procured  for  a  particular 
part  or  component  and  how  are  they  rank  ordered  in  terms  of 
relative  importance?  * 

2.  How  are  the  identified  criteria  combined  to  form  a 
decision-making  model? 

3.  Does  the  model  accurately  determine  when  acquisi¬ 
tion  data  should  be  procured  and  when  it  should  not  be 
procured? 

General  Research  Plan 

This  section  presents  a  brief  overview  of  the  general¬ 
ized  approach  to  the  research. 

There  are  two  basic  approaches  to  answering  the  prob¬ 
lem.  One  method  would  be  to  identify  a  large  sample  of 
procurement  items  for  which  acquisition  data  had  been  pur — 
chased  but  never  used.  Factors  common  to  these  items  and  to 


their  data  purchase  decisions  might  then  be  identified  and 


quantified.  The  difficulty  in  this  approach  is  that  the 
reasons  for  originally  acquiring  the  data,  a  decision  made 
perhaps  as  far  back  as  ten  years  ago,  would  be  difficult  to 
ascertain  solely  from  the  available  documentation.  Also, 
determining  the  effects  of  the  time  factor,  in  the  form  of 
requirement  changes,  on  the  need  for  acquisition  data  for  a 
particular  item  would  require  a  detailed  historical  analysis 
of  each  particular  item  examined.  It  would  also  entail  a 
detailed  historical  analysis  of  its  parent  weapon  system, 
and  the  technology  and  manuf acturing  techniques  applicable 
to  both. 

An  alternative  approach  is  to  assume  that  the  current 
decision-makers  are  attempting  only  to  purchase  acquisition 
data  when  they  can  identify  the  poteptial  for  its  eventual 
use.  The  factors  used  by  the  decision-makers  in  making 
their  decisions  can  then  be  identified  and  formed  into  a 
model.  This  was  the  method  used  in  this  research  effort. 

The  general  research  plan  will  concentrate  on  collect¬ 
ing  and  organizing  the  knowledge  of  a  selected  sample  of 
active  duty  Air  Force  Officers  and  DOD  civilians  concerning 
the  procurement  of  acquisition  data. 

The  research  is  divided  into  three  main  sections,  or 
phases.  Each  phase  will  correspond  to  one  of  the  three 
research  questions.  Each  research  question  will  in  turn  be 

used  to  satisfy  one  of  the  three  research  objectives.  In 

\ 

addition,  each  succeeding  research  objective  will  build  upon 
the  results  of  the  previous  research  objective  (See  Fig.  2). 
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Figure  2.  Research  Plan 


The  first  phase  of  the  research  will  attempt  to  identi¬ 
fy  the  relevant  decision-making  factors  involved  in  the 
procurement  of  acquisition  data  (Research  Objective  #1). 

The  research  vehicle  used  in  this  phase  will  be  a  structured 
interview  (See  Appendix  F) .  The  results  of  this  interview, 
which  is  based  in  part  on  findings  from  the  literature,  will 
be  processed  using  statistical  techniques  (See  Appendix  C> 


to  ensure  that  a  consensus  exists  among  all  individuals 


interviewed  as  to  the  relative  importance  of  the  decision¬ 
making  criteria.  If  all  the  individuals  do  not  agree  after 
the  initial  interviews,  the  Delphi  Technique  (See  Appendix 
A)  will  be  used  to  reach  a  consensus. 

The  decision-making  criteria  concerning  the  procurement 
of  acquisition  data  identified  in  the  first  research  phase 
will  be  used  in  the  second  research  phase  to  form  a  decision 
making  model  (Research  Objective  #2).  The  model  will  be 
based  on  the  Kepner — Tregoe  approach  to  decision-making  (See 
Appendix  B) .  The  Kepner — Tregoe  Model  requires  that  weight 
factors  be  assigned  to  certain  decision-making  criteria. 
These  weights  will  be  identified  using  the  same  structured 
interview  and  consensus  achieving  techniques  that  were 
applied  in  the  first  research  phase. 

The  third  research  phase  involves  testing  the  decision¬ 
making  model  (Research  Objective  #3)  formed  during  the 
second  research  phase.  Testing  will  be  conducted  by 
applying  the  model  to  a  random  sample  of  parts/components 
where  the  decision  to  procure/not  procure  acquisition  data 
is  already  known.  A  comparison  of  the  known  decisions  to 
the  results  of  the  model  applications  will  result  in  a 
success/failure  rate  for  the  model’s  ability  to  make  the 
"correct''  decision  concerning  the  procurement  of  acquisition 


data 


Research  Report  Overview 

As  an  overview  of  this  research  report,  the  content  of 
the  remaining  chapters  is  briefly  discussed. 

Chapter  II  is  a  review  of  the  literature  that  is  appli¬ 
cable  to  the  procurement  of  acquisition  data.  The  first 
area  discussed  is  the  spare  parts  provisioning  process, 
which  ultimately  generates  the  requirements  for  acquisition 
data.  Acquisition  data  is  defined  in  greater  depth  than 
previously,  along  with  a  detailed  description  of  the  process 
by  which  acquisition  data  is  procured.  The  difficulty  with 
the  early  definition  of  acquisition  data  requirements  is 
also  discussed.  Finally,  the  results  of  previous  research 
that  has  been  done  in  this  subject  area  are  outlined. 

Chapter  III  describes  the  methodology  used  to  accom¬ 
plish  the  research  objectives  and  answer  the  research  ques¬ 
tions  discussed  in  Chapter  I.  This  chapter  describes  the 
population  from  which  the  research  data  was  collected,  the 
structured  interview  instruments  that  were  used  to  gather 
the  data,  the  procedures  used  to  process  and  analyze  the 
data,  and  the  techniques  used  to  form  and  validate  the 
decision-making  model.  Finally,  scope,  limitations,  and 
assumptions  of  the  research  are  described. 

Chapter  IV  contains  an  analysis  of  the  data  collected 
from  the  survey  instruments  and  a  description  of  the  model - 
building  and  model  validation  processes.  The  procedures 
described  in  Chapter  III  will  be  used  to  process  the  data 
and  to  build  and  validate  the  model. 


Chapter  V  discusses  the  character! sties  o-f  the 
acquisition  data  decision-making  model  developed  by  the 
researchers,  and  provides  a  greater  explanation  o-f  the 
criteria  and  decision  points  contained  in  the  model.  In 
addition,  this  chapter  explains  how  the  model  is  to  be  used 
in  the  day-to-day  work  environment. 

Finally,  Chapter  VI  summarizes  the  results  drawn  from 
the  research  objectives.  The  utility  of  the  decision-making 
model  is  discussed  along  with  recommendations  for  its  use. 
Recommendations  for  further  research  in  this  area  are  also 


discussed 


II 


Li terature  Review 


Spare  Parts  Provisioning  Process 

The  provisioning  process  enables  the  Air  Force  to 
identify  the  type  and  quantity  of  spare  parts  needed  to 
support  the  initial  fielding  of  an  operational  weapon 
system.  The  process  also  establishes  the  manner  in  which 
spare  parts  will  be  procured  (2:2-21).  Available  procure¬ 
ment  alternatives  include  open  competitive  bidding  or  sole- 
source  procurement  from  the  prime  contractor  (17:2-5). 

Generally,  the  prime  contractor  is  asked  (required  by 
the  contract)  to  recommend  the  method  of  procurement  for  a 
spare  part.  This  is  accomplished  through  the  use  of  numeric 
Contractor  Recommended  Codes  (CRC)  and  suffix  codes  which 
indicate  the  basis  for  the  assignment  of  a  particular  nu¬ 
meric  code  (5:2-6).  A  description  of  the  CRCs  and  suffix 
codes  (See  Appendix  G)  is  contained  in  MIL-STD-789B,  "Pro¬ 
curement  Methods  Coding  of  Replenishment  Spare  Parts  C17:43." 

Following  their  submittal,  the  Contractor  Recommended 
Coders  are  reviewed  by  a  team  of  representati ves  from  the  Air 
Force  Air  Logistic  Center  (ALC)  which  will  have  responsi¬ 
bility  for  the  logistical  support  of  the  weapon  system  under 
consideration  (24:7).  The  review  team  considers  the  CRCs, 
negotiations  with  the  prime  contractor,  and  various  other 
factors  such  as  design  stability  in  assigning  a  Procurement 
Method  Code  (PMC)  and  PMC  suffix  to  the  spare  part  under 
consideration.  The  particular  PMC  and  suffix  assigned 


determine  the  method  by  which  the  spare  part  will  be  pro¬ 
cured  throughout  the  life  of  the  part’s  weapon  systems.  The 
Procurement  Method  Codes  and  Procurement  Method  Code  suf¬ 
fixes  are  also  described  in  MIL-STD-789B  (See  Appendix  S) 
(17:2-11).  Those  items  PMC  coded  for  competitive  procurement 
are  the  items  which  will  require  acquisition  data  (6:5—32). 


The  Nature  of  Acquisition  Data 

Acquisition  and  engineering  data,  defined  in  chapter  I, 
are  actually  specific  forms  of  technical  data.  Technical 
data  is  defined  in  DGD  Instruction  5010.12,  "Management  of 
Technical  Data"  as: 

...recorded  information  used  to  define  a  design 
and  to  produce,  support,  maintain  or  operate  items 
of  defense  material.  These  data  may  be  recorded  as 
graphic  or  pictorial  delineations  in  media  such  as 
drawings  or  photographs;  text  in  specifications  or 
related'  performance  or  design  type  documents;  in 
machine  forms  such  as  punched  cards,  magnetic  tape, 
computer  memory  printouts;  or  may  be  retained  in 
computer  memory.  Examples  of  recorded  information 
include  engineering  drawings  and  associated  lists, 
specifications,  standards,  process  sheets,  manuals, 
technical  reports,  catalogue  item  identifications, 
and  related  information  C153. 

Acquisition  data  is  further  defined  in  MIL-STD-885B, 
"Procurement  Data  Packages. "  According  to  this  document 
acquisition  data  is: 

...a  generic  term  applicable  to  types  of  technical 
data  when  used  for  procurement  purposes.  It  is  a 
composite  of  specifications,  plans,  drawings,  stand¬ 
ards  and  other  data  as  may  be  necessary  to  describe 
existing  materials  so  they  may  be  procured  by  the 
method  contemplated  C16:23. 

Additional  clarification  as  to  what  constitutes  acqui¬ 


sition  data  is  contained  in  MIL-STD-490,  "Specification 


Practices"  which  establishes  the  -format  and  content  of  sys¬ 
tem  specifications,  which  together  with  drawings,  form  a 
basis  for  a  Technical  Data  Package  (TDP)  which  can  be  used 
for  competitive  procurement.  Type  'C*  Product  Specifica¬ 
tions  are  defined  as  specifications  used  in  the  production 
of  a  prime  item  of  equipment  and  are  essentially  sufficient 
to  serve  as  a  TDP.  Specifically,  a  Type  Clb,  Prime  Item 
Product  Fabrication  Specification  contains  all  the  informa¬ 
tion  needed  for  competitive  procurement  when  combined  with 
engineering  drawings  and  associated  lists  (18). 

DOD-IOOOB,  “Drawings,  Engineering  and  Associated  Lists" 
is  the  specification  which  defines  different  levels  of 
drawings  progressing  from  system  inception  to  production. 
Level  3  drawings  provide  engineering  data  for  quantity 
production  of  an  end  item  of  equipment  and  for  the  competi¬ 
tive  procurement  of  spare  parts  which  are  substantially 
identical  to  the  original  items.  If  level  3  drawings  are 
specified  in  a  contract  and  delivered  with  acceptable  quali¬ 
ty  and  unrestricted  rights,  the  Air  Force  should  have  suffi¬ 
cient  information  to  procure  competitively  (5:2-5). 

Acquisition  Data  Procurement  Process 

The  process  of  obtaining  technical  data  (including 
acquisition  data)  begins  within  the  System  Program  Office  at 
the  beginning  of  the  weapon  system  acquisition  cycle.  The 
typical  System  Program  Office  is  a  matrix-type  organization. 


with  a  Program  Manager  being  appointed  during  the  conceptual 


phase  of  the  weapon  system  acquisition  process.  The  Program 
Manager  is  responsible  for  all  aspects  of  the  weapon  system 
acquisition  <29:10).  The  Program  Manager  will  appoint  a  Data 
Management  Officer  (or  serve  in  that  capacity  himself),  who 
serves  as  the  focal  point  for  the  management  of  data  actions 
during  the  life  of  the  program  (4:4-5). 

Prior  to  contract  award,  the  Data  Management  Officer 

issues  a  data  call  to  all  agencies  which  have  a  potential 

need  for  data  (4:4-5).  A  data  call  is  defined  as: 

...a  request  by  the  System/Project  Manager, 
Commander  or  other  authority  to  all  Government 
participants  to  submit  their  requirements  for 
contractor — prepared  data  on  a  given  procurement 
action  CIS]. 

Agencies  that  believe  they  have  a  need  for  data  will  respond 
to  the  data  call  by  identifying  the  type  of  data  and  the 
delivery  schedule  that  they  require.  Requests  for  data  must 
be  justified  by  the  requestors  (24:5). 

After  data  requests  have  been  received  by  the  Data 
Management  Officer,  they  are  reviewed  in  accordance  with  Air 
Force  Regulation  310-1,  "Management  of  Contractor  Data." 

This  review  is  accomplished  by  the  Data  Requirements  Review 
Board,  which  is  chaired  by  either  the  Program  Manager  or  the 
Data  Management  Officer.  This  board  screens  all  requests 
for  data  in  an  attempt  to  eliminate  those  that  are  unneces¬ 
sary.  The  screened  list  of  data  requirements  is  included  in 
the  procurement  contract  as  a  contractor  specification 
through  the  use  of  DOD  Form  1423,  "Contract  Data  Require¬ 
ments  List"  (CDRL) .  The  contractor  is  then  contractual ly 


required  to  provide  the  appropriate  data  according  to  the 
specified  schedule  (4:1-14).  A  pictorial  description  of 
the  technical  data  procurement  process  is  shown  in  Figure  3. 

The  Acquisition  Data  11  Ti me '*  Problem 

The  quandary  in  the  procurement  of  acquisition  data  is 
that  it  is  desirable  to  define  data  requirements  early  in 
order  to  communicate  those  requirements  to  the  contractor 
and  hence,  reduce  total  costs  related  to  engineering  data 
acquisitions.  Vet,  the  Air  Force  must  avoid  specifying  what 
data  is  to  be  acquired  until  requirements  are  actually  known 
and  design  stability  is  achieved  (6:5-31). 

A  recent  study  by  the  Analytics  Corporation,  based  on 
interviews  with*  personnel  fro*  USAF  Aeronautical  Systems 
Division  (ASD) ,  Air  Force  Logistics  Command  (AFLC) ,  and  the 
ALCs,  determined  that  the  earliest  point  in  the  weapon 
system  acquisition  process  where  a  feasible  decision  con¬ 
cerning  acquisition  data  requirements  could  be  made  was 
immediately  after  the  Physical  Configuration  Audit  (6:5-33). 
The  Physical  Configuration  Audit  (PCA)  is  the  formal  exami¬ 
nation  of  the  "as  built"  configuration  of  an  item  against 
its  technical  documentation  or  product  baseline.  PCA  typi¬ 
cally  occurs  when  the  first  production  item  is  produced  (11). 
The  Analytics  study  indicated  that  while  the  PCA  is  the 
earliest  feasible  time  for  a  definition  of  acquisition  data 
requirements,  it  is  "too  late"  in  terms  of  reducing  total 
costs  related  to  engineering  data  (6:531-32). 
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The  Analytics  study  concluded  that  no  clear  criteria 


exist  -for  an  early  definition  of  acquisition  data  require¬ 
ments  concerning  the  procurement  of  spare  parts  (6:5-33). 

In  order  to  define  acquisition  data  requirements  sooner 
in  the  weapons  system  acquisition  process,  the  Analytics 
study  recommended  that  AFLC  examine  the  feasibility  and  cost 
of  developing  profiles  of  typical  spares  requirements  for 
various  types  of  systems  and  equipments.  The  requirements 
for  acquisition  data  could  then  be  based  upon  these  profiles 
(6:6-6).  This  recommendaton  is  roughly  analogous  to  the 
proposals  of  two  other  research  studies  in  this  subject 
area. 

Previous  Research  Studies 

The  two  main  works  on  the  subject  of  decision-making 
criteria  used  for  the  procurement  of  acquisition  data  are 
studies  prepared  by  graduate  students  at  the  Air  Force 
Institute  of  Technology  (AFIT)  located  at  Wright -Patterson 
AFB. 

The  first,  "An  Assessment  of  Relevant  Decision-Making 
Factors  in  the  Purchase  of  Reprocurement  Data,”  was  com¬ 
pleted  in  1779  by  Captain  Robert  L.  Johnson,  USAF,  and 
Captain  Mark  A.  Southwick,  USAF. 

The  objectives  of  Johnson  and  Southwick’ s  thesis  were: 

1.  To  identify  the  factors  considered  in  determining 
whether  reprocurement  data  are  purchased. 

2.  To  provide  general  guidelines  which  will  assist  Air 
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Force  Managers  in  deciding  whether  to  purchase  reprocurement 
data  (24:21). 


The  research  methods  used  by  Johnson  and  Southwick  were 
a  research  questionnaire  and  a  case  analysis  (24:23).  The 
questionnaire  was  used  to  poll  ALC  Engineering  Data  Section 
Personnel  concerning  the  -factors  they  considered  when  decid¬ 
ing  to  procure  acquisition  data  (24:23-24).  The  case  analy¬ 
sis  statistically  analyzed  the  characteristics  of  spare 
parts  that  exhibit  a  high  annual  usage  rate  (AUR).  This 
analysis  was  based  on  the  assumption  that  parts  with  a  high 
AUR  will  need  to  be  procured  on  a  continuing  basis,  and 
hence,  should  be  reprocured  competitively,  i.e.,  would  need 
acquisition  data.  The  weapon  system  used  for  the  case 
analysis  was  the  C-130  aircraft  (24:23,28).' 

Johnson  and  Southwick’s  research  questionnaire  identi¬ 
fied  seven  factors  used  by  Air  Force  Managers  when  deter — 
mining  the  need  for  acquisition  data: 

1.  The  cost  of  the  replenishment  item. 

2.  The  design  stability  of  the  item. 

3.  Expected  life  of  the  item  in  the  inventory. 

4.  The  cost  of  the  reprocurement  data. 

5.  ASPR  (now  FAR)  requirement  to  purchase  competitive¬ 
ly  whenever  possible. 

6.  The  possibility  that  the  original  supplier  may  go 
out  of  business. 

7.  The  lack  of  alternatives  to  promote  competitive 


buying  (24:85) 


In  addition  to  identifying  these  factors,  Johnson  and 
SouthMick  used  a  form  of  factor  analysis  to  determine  that 
the  three  most  important  of  the  above  factors  in  order  were: 

1.  The  design  stability  of  the  item. 

2.  The  cost  of  the  replenishment  item. 

3.  Length  of  expected  life  of  the  item  in  the  inven¬ 
tory  (24:86). 

Johnson  and  Southwick’ s  case  analysis  of  C-130  spare 
parts  identified  a  statistical  correlation  between  the  ac¬ 
tual  method  of  procurement  (AMOP)  of  a  spare  part  (competi¬ 
tive  or  not  competitive)  and  that  part’s  annual  usage  rate 
(AUR)  (24:48).  In  addition,  a  statistical  correlation  was 
identified  between  the  AUR  of  a  spare  part  and  that  part’s 
Item  Classification  (as  defined  by  the  first  two  digits  of 
the  Federal  Stock  number)  (24:87). 

Based  on  these  statistical  correlations  Johnson  and 
Southwick  recommended  that  the  Item  Classification  of  spare 
parts  be  used  as  the  primary  criterion  in  the  decision  to 
procure  acquisition  data  (24:88). 

Johnson  and  Southwick  qualified  their  findings  as  being 
limited  to  the  C-130  aircraft.  They  recommended  that  simi¬ 
lar  studies  be  performed  on  other  weapon  systems  to  deter — 
mine  if  their  decision-making  criterion  had  universal  appli¬ 
cation  (24:90). 

This  recommendation  was  followed  by  Falconer  and  Murphy 
in  their  thesis,  "An  Analysis  of  Factors  which  should  be 
Considered  in  Developing  Repocurement  Data  Requirements." 


Falconer  and  Murphy  applied  the  research  techniques 
used  by  Johnson  and  Southwick  to  other  weapon  systems. 
Specifically,  case  analyses  were  performed  on  the  KC-135 
cargo-type  aircraft,  and  the  F-4  and  A— 7  fighter — type  air¬ 
craft  (21:2). 

Falconer  and  Murphy’s  research  produced  the  following 
findings: 

1.  For  each  aircraft  studied,  a  statistical  correla¬ 
tion  existed  between  the  actual  method  of  procurement  and 
the  annual  usage  rate  of  a  spare  part. 

2.  For  each  aircraft  studied,  a  statistical  correla¬ 
tion  existed  between  the  annual  usage  rate  and  Item  Classi¬ 
fication  of  a  spare  part. 

3.  The  same  Item  Classifications  on  each  weapon  system 
do  not  exhibit  the  same  annual  usage  rate.  This  means  that 
there  is  a  particular  Item  Classification/annual  usage  rate 
relationship  peculiar  to  every  weapon  system.  This  is  a  key 
point,  it  means  that  the  Item  Classif i cation/annual  usage 
rate  relationships  of  preceeding  weapon  systems  cannot  be 
used  to  predict  similar  relationships  of  new  systems  (21:58) 

To  utilize  the  findings  of  these  two  research  studies 
in  the  decision-making  process,  an  analysis  of  every  Air 
Force  weapon  system  would  have  to  be  performed  to  determine 
the  system  peculiar  relationship  between  annual  usage  rate 
and  Item  Classif ication.  This  method  ignores  the  need  for 
some  decision-making  criteria  in  the  early  stages  of  the 
weapon  system  acquisition  process.  In  order  to  economically 


contract  for  acquisition  data,  accurate  decisions  concerning 
the  need  for  acquisition  data  have  to  be  made  before  the 
first  production  item  is  manufactured.  It  is  not  feasible 
to  wait  until  spare  parts  are  procured  to  determine  the  need 
for  acquisition  data  for  those  parts.  The  methodology  in 
Chapter  III  describes  an  attempt  to  provide  decision-making 
criteria  that  can  be  used  in  the  early  stages  of  the  weapon 
system  acquisition  process. 


III.  Methodology 


This  chapter  describes  the  methodology  by  which  the 
research  questions  in  Chapter  I  were  answered. 

Included  in  this  chapter  are  descriptions  of  the 
research  site  (including  a  discussion  of  why  it  was  chosen), 
the  survey  population,  and  the  sample  o-f  individuals  inter — 
viewed.  Next,  the  researchers  discuss  decision-making 
models  and  techniques  in  general,  and  why  the  techniques 
used  in  this  thesis  were  selected.  This  is  -followed  by  a 
description  o-f  the  step-by-step  process  the  researchers  used 
to  answer  all  research  questions  and  objectives.  This 
process  is  broken  down  into  three  main  Research  Phases,  with 
each  phase  corresponding  to  one  Research  Objective.  A 
flowchart  accompanies  each  phase  to  allow  for  easier  under — 
standing  of  the  methodology  used  by  the  researchers.  The 
researchers  conclude  by  listing  all  assumptions  and  limita¬ 
tions  pertaining  to  the  methodology,  and  provide  the  reader 
with  a  brief  summary  of  the  methodology,  including  a  flow¬ 
chart  of  the  entire  research  methodology. 

Related  to,  but  not  included  in,  the  methodology  are 
descriptions  of  the  Delphi  Technique  (Appendix  A),  the 
Kepner — Tregoe  decision-making  analysis  (Appendix  B) , 
Kendall’s  Coefficient  of  Concordance,  W  (Appendix  C> ,  and 
the  test  for  Differences  Between  Two  Population  Means 
(Appendix  D) .  Appendix  E  contains  abbreviated  Chi-Square 
and  t-test  tables  used  under  the  "Kendall  Coefficient"  and 


“Difference  Between  Two  Means"  tests.  Finally,  Appendix  F 
contains  a  copy  of  the  two  structured  interview  question¬ 
naires  used  to  collect  the  data  for  this  thesis. 

Research  Site 

This  thesis  effort  surveyed  Program  Managers  and  Con¬ 
figuration  Management  Personnel  within  AFSC’s  Aeronautical 
Systems  Division  (ASD)  located  at  Wright-Patterson  AFB, 

Ohio.  ASD  was  chosen  because  it  is  located  at  the  same  site 
as  the  researchers  and  this  organization  purchases  78%  of 
all  acquisition  data  within  the  U.S.  Air  Force  (26).  Having 
the  research  site  located  on  the  same  base  allowed  the 
research  team  to  collect  data  within  the  time  constraints  of 
the  thesis  completion  date. 

Population 

The  survey  papulation  consisted  of  all  civilian  or 
military  personnel  who  participate  in  the  Procurement  Method 
Coding  process  at  AFSC/ASD.  The  total  population  consists 
of  approximately  256  people.  To  determine  the  survey 
population  size,  the  researchers  relied  on  information 
supplied  by  the  Directors  of  each  program  office  at  ASD. 

Samp 1 e 

The  survey  sample  was  drawn  from  the  list  of  individu¬ 
als  supplied  by  the  program  office  directors.  To  make  the 
assumption  that  the  selected  sample  accurately  represents 
each  program  office  within  ASD,  a  sample  size  of  thirty 


individuals  was  chosen.  In  order  to  invoke  the  Central 


Limit  Theorem,  a  sample  size  of  at  least  twenty  individuals 
should  have  been  selected.  The  greater  the  sample  size,  the 
better  the  approximation  of  the  population.  The  researchers 
felt,  however,  that  a  sample  greater  than  thirty  would  be 
too  unwieldy,  given  the  time  constraints  placed  on  thesis 
completion. 

Using  the  Central  Limit  Theorem  requires  that  an 
assumption  be  made  that  no  respondents  discuss  interview 
responses  with  other  respondents.  This  assumption  may  be 
supported  because  the  researchers  warned  each  respondent 
that  discussing  their  responses  with  others  will  invalidate 
any  results  obtained  from  1  structured  interviews. 

The  exact  number  of  persons  from  each  program  office 
was  chosen  using  a  stratified  random  sampling  technique  in 
conjunction  with  the  following  method: 

TABLE  I 

Calculation  of  the  Number  of  Representatives  Interviewed  * 


ORG’N 

(POP 

X 

1 1 . 727. 

53 

AMT) 

SAMPLE 

AE 

31 

X 

1 1 . 727. 

s 

3.63 

4 

AF 

16 

X 

1 1 . 727. 

s 

1.88 

2 

B— 1 

12 

X 

1 1 . 727. 

sc 

1.41 

1 

RW 

77 

X 

1 1 . 727. 

ss 

9.02 

9 

YP 

12 

X 

1 1 . 727. 

s 

1.41 

1 

YY 

4 

X 

1 1 . 727. 

s 

0.47 

1 

YZ 

25 

X 

1 1 . 727. 

s 

2.93 

3 

TA 

79 

256 

X 

1 1 . 727. 

= 

9.26 

9 

30 

*  At  least  one  person  was  required  from  each  SPO 
(except  for  those  SPOs  with  no  data  personnel ) 


1.  A  percentage  of  sample  size  to  population  size  Mas 
calculated  —  30/256  -  11.72*/.. 

2.  This  percentage  was  applied  to  the  total  number  of 
data  personnel  available  within  each  program  office.  Table 
I  shows  the  exact  number  of  individuals  selected  from  each 
program  office. 

Those  individuals  selected  by  their  supervisors  were 
deemed  “experts"  in  the  area  of  data  acquisition.  Because 
it  relied  on  the  respondents  being  "experts*1  in  their  field, 
this  methodology  precluded  random  selection  (23:11). 

Problem  Solving  Approach 

Decision-Making  Model s  and  Techniques.  There  are  a 
number  of  decision  theory  approaches  available  to  today’s 
manager.  Most  techniques,  such  as  the  zero-sum  and  the 
mixed  strategy  games,  use  the  concepts  of  competitors’ 
actions  and  states  of  nature.  These  concepts  did  not  apply 
to  the  present  situation  of  determining  when  to  purchase 
acquisition  data;  there  are  no  d4  fferent  outcomes  possible 
in  the  states  of  nature  —  you  either  choose  to  purchase 
acquisition  data  or  you  do  not.  Neither  is  there  a  known 
penalty  for  not  choosing  one  alternative  over  another. 

According  to  Schoderbek,  Schoderbek  and  Kef alas, 
"probably  the  most  fundamental  element  in  COper at ions 
Research!  is  the  need  to  quantify  the  business  problem  under 


study  C  3 1 :  1 33  3 " . 


However,  decisions  that  are  more  o-f  a  judgmental 
nature,  less  prone  to  recur  with  regularity,  and  more 
affected  by  environmental  -Factors  are  not  readily 
subject  to  quant i-Fi cation  C31:1333. 

Because  the  researchers  were  working  with  subjective 
decision  criteria,  Bayesian  Decision  Criteria,  Statistical 
Decision  Theory,  and  other  Operations  Research  techniques 
did  not  apply.  Instead,  the  researchers  attempted  to  -Find  a 
technique  which  would  lead  to  “Go”  and  “No  Go"  decision 
points  based  on  established  criteria.  Since  no  criteria 
were  presently  set  -Forth  under  any  regulation,  publication, 
or  guide,  the  researchers  -First  needed  to  determine  criteria 
which  would  aid  in  the  acquisition  data  purchase  decision. 
The  criteria  presently  used  are  "wisdom,  luck,  and  a  good 
measure  o-F  experience  Z26\.  " 

One  desirable  way  to  determine  objectives  or  criteria 
(when  none  are  already  set  -Forth)  is  to  organize  members 
into  panels  or  committees  (8:191).  The  intent  is  to  tap  the 
knowledge  available  from  within  a  group  and  arrive  at  a 
consensus.  However,  there  are  several  problems  inherent  in 
gathering  data  through  group  meetings,  such  as  the  influence 
of  dominant  members  or  the  pressure  to  conform.  The  Delphi 
technique  was  chosen  as  the  means  to  extract  the  expertise 
and  experience  from  those  individuals  who  have  been  working 
in  the  area  of  purchasing  acquisition  data,  because  it 
overcomes  the  inherent  weaknesses  of  grotio  sessions.  For  a 
complete  explanation  of  the  Delphi  Technique,  refer  to 


Appendix  A 


Because  many  of  the  criteria  found  in  the  literature 
review  were  quite  subjective  in  nature,  quantitative 
decision  theory  models  were  of  no  use  (20).  The  only  suit¬ 
able  decision-making  technique  which  utilizes  both  subjec¬ 
tive  criteria  and  subjective  ratings  of  those  criteria  is 
the  Kepner-Tregoe  approach  (25).  Appendix  B  contains  a 
thorough  description  of  this  decision-making  tool. 

The  Process.  Figure  4  takes  the  reader  through  Steps 
#1,  #2  and  #3  of  the  methodology  to  accomplish  Research 
Objective  I.  Figure  5  depicts  the  problem-solving  approach 
to  achieve  Research  Objective  II,  and  includes  Steps  #4 
through  #9.  Finally,  Figure  7  shows  the  approach  taken  to 
attain  Research  Objective  III,  and  it  includes  Steps  #10  and 
#11  of  the  research  methodology. 

Step  #1.  The  researchers  used  a  "structured'1'  interview 
to  obtain  acquisition  data  decision  criteria  from  the  sam¬ 
ple.  A  structured  interview  is  very  similar  to  a  survey 
technique  in  that  all  questions  are  prepared  prior  to  the 
interview;  there  is  no  spontaneous  exchange  of  ideas  between 
the  interviewer  and  the  respondent. 

The  survey  instrument  (See  Appendix  F)  contained  a  list 
of  all  criteria  found  by  the  researchers  through  their 
literature  review,  along  with  a  brief  explanation  of  the 
criteria.  The  respondents  were  asked  to  rate  each  criterion 
on  a  scale  from  zero  to  five  based  on  its  importance  in 
deciding  whether  or  not  to  purchase  acquisition  data.  If 
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Figure  4.  Methodology  Flowchart  —  Phase  I 

the  criterion  was  very  important,  it  should  have  been  given 
a  rating  of  five;  if  it  was  not  at  all  significant,  it 
should  have  been  given  a  rating  of  zero.  Any  time  a  respon¬ 
dent  rated  a  criterion  as  a  zero  or  a  five,  rationale  for 
the  decision  was  requested.  If  the  researchers  could  not 
have  shown  consensus  on  the  ranking  of  the  criteria  (see  the 
end  of  this  subsection  for  a  discussion  on  ranking  the  cri¬ 
teria),  these  rates  would  have  been  used  to  calculate  inter¬ 
quartile  ranges  and  mean  ratings.  This  data  would  then  have 
been  provided  as  "feedback"  under  a  second  set  of  interviews 
to  determine  the  decision-making  criteria. 


In  addition,  the  respondent  was  asked  to  determine 
whether  the  criterion  is  a  "MUST”  or  a  "WANT."  A  "MUST" 
criterion  is  one  which  absolutely  must  be  present  before 
acquisition  data  will  be  purchased.  If  this  criterion  does 
not  exist,  the  Government  will  not  purchase  the  acquisition 
data.  A  "WANT"  criterion  is  one  which  is  desirable  when 
deciding  whether  or  not  to  purchase  acquisition  data,  but 
is  not  absolutely  necessary. 

Because  many  of  the  potential  criteria  listed  on  the 
questionnaire  were  rather  ambiguous  (design  stability,  item 
complexity,  etc.),  respondents  were  also  asked  to  provide 
their  opinions  of  how  these  criteria  are  defined.  In 
addition,  respondents  were  allowed  to  add  any  criteria  not 
already  listed  on  the  questionnaire.  Finally,  the  respon¬ 
dents  were  asked  to  rank  order  all  criteria  (including 
criteria  thay  have  added),  from  most  important  to  least 
important.  Kendall’s  Coefficient  of  Concordance  requires 
that  the  "experts"  rank  order  all  criterion  before  concensus 
can  be  determined  (See  Step  #2). 

Step  #2.  In  this  step  the  researchers  tested  for 
consensus  among  respondents  as  to  the  ranking  of  criteria 
obtained  from  Step  #1.  The  researchers  used  Kendall’s 
Coefficient  of  Concordance,  W,  to  test  for  overall  agreement 
among  the  respondents.  A  complete  description  of  this  test 
is  provided  in  Appendix  C. 


Step  #3.  This  step  simply  entailed  ranking  all  the 
criteria  -found  under  Step  #1  in  order  of  importance  -from 
most  important  to  least  important.  The  overall  rankings 
were  based  on  the  sums  o-f  the  ranks  -for  each  criterion.  The 
criterion  with  the  lowest  s’m  was  ranked  number  one  in 
importance.  The  criterion  with  the  next  lowest  sum  was 
ranked  number  two  in  importance,  and  so  on,  until  the 
criterion  with  the  highest  sum  remained.  This  criterion  was 
ranked  lowest  among  all  the  decision-making  criteria  provided 
by  the  acquisition  data  experts. 

This  concludes  Phase  I  of  the  research  effort  and 
provides  the  reader  with  the  answer  to  Research  Objective  I, 
“To  identify  and  rank  all  criteria  .  .  .  .“  Figure  5 
depicts  the  methodology  for  solving  Phase  II,  Research 
Objective  1 1  of  this  research  effort. 

Step  #4.  Because  the  researchers  found  that  no  ,,liUST,, 
criteria  existed,  all  eleven  criteria  had  to  be  weighed 
using  a  second  “structured  interview"  technique.  The  survey 
instrument  for  determining  the  criteria’s  weights  can  be 
found  in  Appendix  F. 

The  researchers  asked  the  respondents  to  weight  each 
criterion  from  one  to  ten  based  on  that  criterion’s  impoi — 
tance  in  the  decision  to  purchase  acquisition  data.  The 
respondents  were  told  that  no  criterion  could  be  weighted 
higher  than  the  previous  criterion,  but  it  could  be  weighted 
the  same  as  the  previous  one.  This  was  to  avoid  the  possi- 
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bility  of  one  criterion  being  ranked  lower  than  another 
after  Steps  #1  through  #3,  and  yet  being  weighted  higher 
after  Step  #4. 

The  first  criterion  in  rank  did  not  necessarily  have 
to  be  given  a  weight  of  ten  <the  highest  possible  weight), 
nor  did  the  last  criterion  have  to  be  given  a  weight  of  one 
(unless,  of  course,  the  previous  criterion  was  given  the 
weight  of  one). 

For  example,  say  that  the  following  criteria  are  ranked 
from  most  significant  to  least  significant: 


1.  Design  stability 

2.  Expected  inventory  life  of  the  item 

3.  Complexity  of  design 

4.  Cost  of  the  acquisition  data 

5.  Amount  of  critical  materials 

6.  Cost  of  the  replenishment  item 

7.  Amount  of  fixed  tooling 

Each  respondent  was  then  asked  to  weight  each  criterion  from 
one  to  ten,  with  ten  being  the  highest.  This  is  how  the 
weightings  might  look  for  one  respondent: 


1.  Design  stability  (9) 

2.  Expected  inventory  life  <9> 

3.  Complexity  of  design  (7) 

4.  Cost  of  the  acquisition  data  (4) 

5.  Amount  of  critical  materials  (2) 

6.  Cost  of  the  replenishment  item  (2) 

7.  Amount  of  fixed  tooling  (2) 


A  second  respondent  may  have  weighted  the  criteria 
differently: 


1.  Design  stability  (lO) 

2.  Expected  inventory  life  (8) 

3.  Complexity  of  design  (7) 

4.  Cost  of  the  acquisition  data  (7) 

3.  Amount  of  critical  materials  (2) 

6.  Cost  of  the  replenishment  item  (1) 

7.  Amount  of  fixed  tooling  (1) 


The  researchers  then  derived  an  average  weight  for  each 
of  the  criteria  using  the  following  equation  for  calculating 
the  arithmetic  mean  of  a  sample  (7:55-59): 


n  <1> 

where:  n  =  total  number  of  respondents 

x  =  the  weight  given  the  criterion  by  respondent  i 


In  the  example  above,  had  only  two  respondents  been 
interviewed,  the  weights  for  the  above  criteria  would  be: 


1.  Design  stability  (9.5) 

2.  Expected  inventory  life  (8.5) 

3.  Complexity  of  design  (7.0) 

4.  Cost  of  the  acquisition  data  (5.5) 

5.  Amount  of  critical  materials  (2.0) 

6.  Cost  of  the  replenishment  item  (1.5) 

7.  Amount  of  fixed  tooling  (1.5) 


Step  #5.  The  researchers  then  had  item  managers  select 
one  item  each  for  which  acquisition  data  was  purchased  and 
later  used  under  a  competitive  bid.  The  item  managers  were 
also  asked  to  select  one  item  each  for  which  no  acquisition 
data  was  purchased. 

Step  #6.  The  item  managers  were  to  rate  the  item  they 
had  selected  using  the  criteria  developed  in  Steps  #1 
through  #4.  The  respondent  rated  the  item  from  one  to  ten 
against  every  criterion,  and  the  rating  was  multiplied  by 

a 

the  weight  for  that  criterion.  The  products  for  each  cri¬ 
terion  were  then  summed,  and  an  "item  value1*  was  obtained 
for  that  particular  item.  If  the  item  being  rated  was  an 
item  for  which  the  acquisition  data  was  purchased  and  used, 
the  item  value  should  have  been  relatively  high.  If  the 
item  being  rated  was  one  for  which  no  acquisition  data  was 
purchased,  the  item  value  should  have  been  relatively  low. 

In  the  example  presented  above,  one  selected  item  may 
have  a  very  high  design  stability,  so  the  respondent  might 
give  it  a  rating  of  8.  It  may  also  be  anticipated  that  this 
item  will  have  a  long  inventory  life,  so  the  respondent  gave 
it  a  rating  of  7.  The  respondent  may  know  that  the  item  is 
somewhat  non— complex,  so  he  gave  it  a  rating  of  8  for 
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"complexity  of  design,"  and  so  on,  until  the  -final  rating 
looks  like  this: 


CRITERION 

WT 

X 

RT 

=  AMT 

1. 

Design  stability 

(9.5) 

X 

9 

=  85.5 

2. 

Expected  inventory  life 

(8.5) 

7 

59.5 

O  • 

Complexity  of  design 

(7.0) 

8 

56.0 

4. 

Cost  of  the  acquisition  data 

(5.5) 

4 

22.0 

5. 

Amount  of  critical  materials 

(2.0) 

8 

16.0 

6. 

Cost  of  the  replenishment  item 

(1.5) 

10 

15.0 

7. 

Amount  of  fixed  tooling 

(1.5) 

7 

10.5 

ITEM  VALUE 

264.5 

Step  #7.  The  researchers  then 

calculated 

an  average 

item  value  o-f  those  items  -for  which  acquisition  data  was 
purchased  and  used,  and  o-f  those  items  for  which  acquisition 
data  was  not  purchased.  Equation  (1)  used  in  Step  #4  to 
calculate  the  average  weight  was  also  used  here  to  calculate 
the  average  item  value. 

The  average  item  value  of  items  for  which  acquisition 
data  was  purchased  and  later  used  (we  call  this  the  "high 
item  value")  became  the  "Go"  decision  point  in  the  final 
model.  The  average  item  value  of  items  for  which  no  acqui¬ 
sition  data  was  purchased  (we  call  this  the  "low  item 
value")  became  the  "No  Go"  decision  point. 

During  operational  use  of  this  decision-making  tech¬ 
nique,  if  a  data  manager  derives  an  item  value  equal  to  or 
greater  than  the  "high  item  value,"  he  should  decide  to 
purchase  acquisition  data.  If  the  calculated  item  value  is 
equal  to  or  less  than  the  "low  item  value,"  he  should  decide 
to  not  purchase  acquisition  data  for  that  item.  If  the  item 
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value  falls  between  the  two  decision  points,  it  is  the  mana¬ 
ger’s  decision  whether  or  not  to  purchase  acquisition  data. 

Step  #8.  In  order  to  infer  that  there  exists  a  true 
difference  between  the  two  decision  points,  the  researchers 
performed  a  test  for  the  difference  between  two  means. 

The  necessary  assumptions  that  1)  both  sampled  populations 
have  relative  frequency  distributions  that  are  approximately 
normal,  2)  the  population  variances  are  equal,  and  3)  the 
samples  are  randomly  and  independently  selected  from  the 
populations,  appeared  to  be  reasonable. 

The  Null  Hypothesis  was  that  the  mean  for  population  1 
equals  the  mean  for  population  2,  while  the  Alternative 
Hypothesis  was  that  the  means  of  the  two  papulations  are  not 
equal.  To  determine  if  the  two  means  are  the  same,  the 
following  values  must  be  calculated  (7:337-342): 

1.  x  -  high  item  value 

1 

2.  x  =  low  item  value 

2 

3.  s  =  standard  deviation  of  high  item  value 

1 

4.  s  »  standard  deviation  of  low  item  value 

2 

5.  sz  -  pooled  sample  estimator  of  52 

P 

6.  t  =»  Student’s  t-statistic 

The  equations  used  to  determine  sj  and  t  are  given 
on  the  following  page: 


Referring  to  TABLE  XII  in  Appendix  E,  the  researchers 
used  a  significance  level  of  *  .05  at  C(3  +  3  -  2)  =43 
degrees  freedom.  If  the  calculated  t-value  exceeds  the  t- 
value  in  Appendix  E,  the  researchers  would  reject  the  Null 
Hypothesis  that  the  high  item  value  is  no  different  than  the 
low  item  value.  In  other  words,  there  is  a  significant 
difference  between  the  "Go"  and  MNo  Go"  decision  points. 

Since  a  significant  difference  was  found  between  the 
high  and  low  item  values,  the  researchers  then  proceeded  to 
Step  #9,  development  of  the  final  model.  For  a  complete 
explanation  of  the  Small-Sample  Test  for  the  Difference 
Between  Two  Population  Means,  see  Appendix  D. 

Step  #9.  In  effect,  the  final  model  has  already  been 
developed  for  the  researchers.  Final  development  merely 
entailed  putting  the  model  into  a  format  which  was  both 
functional  and  aesthetic.  Figure  6  is  the  final  model 
format  chosen  by  the  researchers.  The  final  model  lists 
each  criterion,  followed  by  its  calculated  weight.  It  would 
be  the  user’s  role  to  fill  in  a  rate  for  each  criterion  and 
find  the  product  of  the  weight  and  the  rate.  The  user  would 
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Figure  7.  Methodology  Flowchart  —  Phase  III 

Step  #10.  To  test  the  -final  model,  the  researchers 
randomly  selected  items  -from  various  program  o-f-fices  and 
had  Program  lanagers  perform  the  decision-making  analysis 
(In  reality,  the  decision  had  already  been  made  to  purchase 
or  not  purchase  acquisition  data  for  these  items).  The 
Program  Manager  calculated  a  Total  Item  Value  for  the  item, 
and,  based  on  the  "Go"/"No  Go"  decision  points,  decided 
whether  or  not  to  purchase  acquisition  data.  His  decision 
was  then  compared  with  the  actual,  documented  decision,  and 
a  total  percentage  of  "correct"  decisions  was  calculated. 

Step  #11.  Conclusions/recommendations  will  be  set 


forth  under  Chapter  VI  of  this  research  effort 


Assumptions 

This  research  effort  was  performed  under  the  following 
assumptions: 

1.  The  best  Data  Managers  were  indeed  chosen  for  their 
abilities  as  data  managers. 

2.  Because  the  researchers  insisted  that  no  informa¬ 
tion  be  discussed  outside  the  interview,  anonymity  was  main¬ 
tained  by  all  respondents  during  the  successive  iterations 
of  the  interviews. 

3.  The  “experts"  will  tend  to  rate  each  item  somewhat 
uniformly. 

4.  Under  those  contracts  for  which  the  Government 
purchased  and  later  used  acquisition  data,  valid  reasons  or 
criteria  existed  for  purchasing  the  data. 

5.  Under  those  contracts  for  which  acquisition  data 
was  not  purchased,  valid  reasons  or  criteria  existed  for  not 
purchasing  the  data. 

Limitations 

1.  This  research  effort  was  limited  to  a  fifteen  month 
schedule  for  completion  as  dictated  by  the  Air  Force  Insti¬ 
tute  of  Technology  (AFIT) ,  School  of  Systems  and  Logistics. 

2.  Personal  interviews  were  limited  in  time  due  to  the 
heavy  workloads  of  the  respondents. 

Summary  of  Methodology 

Figure  8  depicts  the  entire  methodology  used  to  develop 
an  acquisition  data  decision-making  model: 
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Figure  8.  Overall  Methodology  Flowchart 


IV.  Data  Analysi s 


The  purpose  of  this  chapter  is  to  collate,  analyze, 
and  present  the  interview  data  using  the  research  methodol¬ 
ogy  that  was  developed  in  Chapter  III. 

This  chapter  has  three  main  sections,  with  each  sec¬ 
tion  corresponding  to  one  o-f  the  three  research  phases  as 
outlined  in  Chapter  I.  In  turn,  each  Research  Phase  is 
designed  to  answer  one  o-f  the  three  thesis  Research  Ques¬ 
tions.  Each  Research  Phase  is  further  subdivided  into  a 
series  of  steps  as  outlined  in  Chapter  III. 

Research  Phase  I 

Research  Phase  I  was  designed  to  answer  Research  Ques¬ 
tion  #1  which  stated:  What  criteria  should  be  considered  in 
determining  whether  acquisition  data  will  be  procured  for  a 
particular  part  or  component,  and  how  are  the  criteria  rank 
ordered  in  terms  of  relative  importance? 

Step  #1 .  The  results  o-f  the  first  structured  interview 
(See  Appendix  F  for  a  copy  of  the  first  survey  tool)  are 
contained  in  Tables  IV  and  V.  Table  III  lists  each  deci¬ 
sion  criteria  beside  its  "criteria  designator."  Table  IV 
depicts  the  experts’  subjective  ratings  of  each  decision 
criterion  on  a  scale  of  1  to  5.  Criteria  that  were  identi¬ 
fied  by  each  expert  as  "MUSTS"  are  indicated  by  the  letter 
"M"  following  each  rating. 


In  order  to  be  considered  a  ,,hUST,,  for  the  purpose  of 
the  decision-making  model,  a  particular  criterion  had  to  have 
been  identified  as  a  "MUST"  by  907.  (27  of  30)  of  the  experts 
interviewed.  The  researchers  chose  the  level  of  90 7.  based 
on  the  opinions  of  ASD  contracting  personnel.  Table  II 
shows  the  results  of  each  contracting  individual’s  opinion 
as  to  how  many  respondents  should  identify  a  particular 
criterion  as  a  “MUST"  before  it  should  be  considered  as  such 
in  the  final  model. 

The  researchers  used  the  median  and  the  mode  as  the 
methods  to  obtain  a  final  level  of  agreement  among  the 
contracting  personnel’s  opinions.  The  median  is  often  a 
better  measure  of  central  tendency  because  it  is  less  sensi¬ 
tive  than  the  mean  to  a  few  extremely  large  or  extremely 
small  measurements  (7:61).  The  mode  is  useful  as  a  measure 
of  central  tendency  because  it  emphasizes  data  concentration 
(7:54).  The  researchers  also  include  the  mean  of  the  sample 
for  comparison  purposes. 

TABLE  II 

Contracting  Personnel’s  Opinions  as  to  the  Level  Required 
Before  a  Criterion  may  be  Considered  a  "MUST" 

Person  #  Percentage  Person  #  Percentage 


Analysis  o-f  the  data  indicated  that  no  decision  cri¬ 


terion  qualified  as  a  "MUST".  Table  IV  depicts  the  total 
number  of  times  that  each  criterion  was  identified  as  a 
"MUST"  by  the  experts  and  it  includes  the  computed  percen¬ 
tages.  Since  the  highest  rating  of  “MUSTS"  for  any  cri¬ 
terion  was  only  40%,  no  criteria  were  designated  as  "MUSTS 
in  the  final  decision-making  model. 

The  letters  under  the  "Criteria  Designator"  column  of 
Table  III  are  used  in  the  "Criteria"  row  of  Table  IV  to 
represent  each  associated  criterion. 

TABLE  III 

List  of  Acquisition  Data  Decision-Making  Criteria 


CRITERIA  CRITERIA 

DESIGNATOR  _ _ . _ 

A  Cost  of  the  Replenishment  Item 

B  Design  Stability 

C  Expected  Inventory  Life 

D  Cost  of  the  Acquisition  Data 

E  Organizational /Regulatory  Requirements 

Original  Supplier’s  Business  Solvency 

Amount  of  Fixed  Tooling 

Anticipated  Quantity  of  the  Buy 

Amount  of  Critical  Materials 


Time/Personnel  Requirements  (Cost  Effectiveness) 
Complexity  of  Design 


Selected  comments  by  the  experts  concerning  the  defini¬ 
tion  of  each  decision  criterion,  and  the  experts’  reasons 
for  assigning  either  the  highest  or  lowest  rating  to  a 
criterion  are  contained  in  Appendix  H. 

In  responding  to  the  first  structured  interview  the 
experts  failed  to  identify  any  new  decision  criteria  that 
would  affect  the  model.  This  was  attributed  to  the  fact 
that  the  eleven  previously  identified  decision  criteria  were 
largely  the  results  of  two  recent  research  efforts  that 
utilized  a  similar  research  population.  In  addition,  other 
relevant  criteria  were  found  through  preliminary  interviews 
with  personnel  knowledgeable  in  the  acquisition  data  envi¬ 
ronment. 

Table  V  depicts  the  individual  experts’  rank  ordering 
of  the  decision  criteria.  This  rank  ordering  by  the  experts 
was  required  in  order  to  test  for  a  consensus  as  to  the 
ranking  of  the  decision  criteria  in  terms  of  relative 
importance.  The  actual  test  for  consensus  is  performed  in 
Step  #2. 

Step  #2.  In  this  step  Kendall’s  Coefficient  of  Concor — 
dance  was  used  to  test  for  consensus  among  the  experts  as  to 
the  ranking  of  the  decision  criteria  in  terms  of  their 
relative  importance.  The  actual  computations  involved  in 
the  analysis  are  presented  following  Table  V.  For  a  more 
detailed  explanation  of  Kendall’s  Coefficient  of  Concordance 
see  Appendix  C. 
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M  Ul 


W  =  Kendall’s  Coefficient  of  Concordance  (0  <  W  <  1) 
k  =  the  number  of  experts  C303 
N  53  the  number  of  criteria  C 1 1 3 

s  =  the  sum  of  the  observed  deviations  from  the  mean 
of  the  SUMS  (SUMS  obtained  from  TABLE  IV) 


2  2  2 
s  *  (132-176.46)  +  (77-176.46)  +  (115-176.46)  + 

2  2  2 

(146-176.46)  +  (231-176.46)  +  (249-176.46)  + 
2  2  2 

(262-176.46)  +  (140-176.46)  +  (225-176.46)  + 
2  2 

(232-176.46)  +  (131-176.46)  *  40876.70 


W  * _ s _ ■  ■  40876.70  =  .41290  (5) 

(1/12)  (k  X  k)  (Ns  -  N)  99000 


Chi-Square  Value  =  (k)(N-l)(W>  ■  30(10). 41290  =  123.869  (6) 


The  test  chi-square  value  at  a  3  .05  level  of  significance 
based  on  10  (N-l)  degrees  of  freedom  33  18.3070. 


Since  the  actual  Chi-Square  value  of  124.134  exceeded 
the  test  value  of  18.3070,  the  null  hypothesis  could  easily 
be  rejected.  Thus,  the  surveyed  experts  achieved  consensus 
in  their  ranking  of  the  decision  criteria  in  terms  of  rela¬ 
tive  importance. 


Step  #3.  As  a  result  of  the  rankings  provided  by  the 
experts,  Table  VI  displays  the  final  ranking  of  the  criteria 


obtained  from  the  first  set  of  interviews.  The  final  rank- 
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ings  are  based  on  the  sums  in  Table  V,  with  the  criterion 
having  the  lowest  sum  ranked  -first,  the  criterion  with  the 
next  lowest  sum  ranked  second,  and  so  on. 

TABLE  VI 

Final  Rank  Ordering  of  Decision  Criteria 


PREVIOUS 

RANK  CRITERIA  DESIGNATOR 

1  Design  Stability  B 

2  Expected  Inventory  Life  C 

3  Complexity  of  Design  K 

4  Cost  of  the  Replenishment  Item  A 

5  Anticipated  Quantity  of  Buy  H 

6  Cost  of  the  Acquisition  Data  D 

7  Amount  of  Critical  Materials  I 

8  Organizational /Regulatory  Requirements  E 

9  Time/Personnel  Requirements  J 

10  Original  Supplier’s  Business  Solvency  F 

11  Amount  of  Fixed  Tooling  G 


NOTE:  From  this  point  forward,  all  criteria  will  be  shown 
using  their  ranks  as  criteria  designators. 

Research  Question  #1  Answered 

With  the  completion  of  step  #3,  the  researchers  have 
now  provided  the  answer  to  Research  Question  #1.  Table  VI 
reflects  the  researchers’  analysis  of  the  data  gathered  from 
the  survey  of  the  experts.  This  data  indicates  which  cri¬ 
teria  should  be  considered  in  determining  whether  acquisi- 
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tion  data  should  be  purchased  for  a  particular  part  or 
component,  ranked  in  relative  order  of  importance.  For  a 
more  detailed  understanding  of  each  of  the  criteria,  see 
Chapter  V  and  Appendix  H. 


TABLE  VII 

Experts’  Subjective  Weighting  of  Decision  Criteria 


CRITERIA 


TOTALS  285  267  249  223  201  182  153  132  110  75  55 


Research  Phase  1 1 


Research  Phase  II  was  designed  to  answer  Research 
Question  #2,  which  stated:  How  are  the  identified  criteria 
combined  to  form  a  decision-making  model? 

Step  #4.  With  no  identified  decision  criteria  being 
qualified  as  a  "MUST",  all  eleven  criteria  were  included  in 
the  next  structured  interview  as  "WANTS".  This  interview 
involved  the  subjective  weighting  of  each  criterion  by  the 
same  group  of  experts.  Weighting  was  done  on  a  scale  of  1 
to  10  in  accordance  with  the  rules  as  described  in  Chapter 
III,  Step  #4.  The  average  weight  of  each  criterion  was  then 
computed  using  the  simple  arithmetic  mean  of  the  responses. 
The  results  of  this  process  can  be  seen  under  Table  VII  in 
the  “AVG  WT"  row. 

Step  #5.  Three  ALC  item  managers  selected  one  item 
each  for  which  acquisition  data  had  been  purchased  and  later 
used  under  a  competitive  bid.  These  items  will  be  referred 
to  as  "High  Item  Value"  items.  They  consisted  of: 

Item  Al:  A  twelve-beam  cathode  Ray  Tube  used  in  an 

imaging  system  for  tactical  reconnaissance. 

Item  A2:  An  airtight  Me Donne 1 1 -Douglas  F-4  Phantom  II 
viewfinder  lens. 

Item  A3:  A  tailpipe  clamp  far  the  Lockheed  T-33. 

Three  ALC  item  managers  also  selected  one  item  each  for 
which  no  acquisition  data  had  been  purchased.  These  items 
will  be  referred  to  as  "Low  Item  Value"  items  and  they 


consisted  of: 


Item  Bl:  Production  carbon  wheels  and  brakes  -for  the 
General  Dynamics  F-16. 

Item  B2:  The  ACES  II  ejection  seat. 

Item  B3:  The  Aim-9P  Sidewinder  Missile. 

Step  #6.  The  six  ALC  item  managers  proceeded  to 
evaluate  each  o-f  the  selected  purchase  items  using  the 
decision  criteria  that  were  identified  in  Research  Phase  I. 
Each  purchase  item  was  assigned  a  rating  (on  a  scale  o-f  1  to 
10)  -for  each  decision  criterion  based  on  the  rater’s  judg¬ 
ment  o-f  how  much  the  item  exhibited  the  traits  o-f  that 
decision  criterion.  The  rate  assigned  to  each  decision 
criterion  was  then  multiplied  by  the  previously  computed 
subjective  weight  (See  Table  VII,  “AVG  WT")  for  that  cri¬ 
terion.  The  resulting  products  were  then  summed  to  obtain 
a  "Total  Item  Value"  for  each  purchase  item.  The  ratings 
and  Total  Item  Values  for  each  purchase  item  are  shown  in 
Table  VIII. 

Step  #7.  The  decision  points  for  the  model  were  then 
calculated  by  computing  the  arithmetic  mean  of  each  set  of 
Total  Item  Values.  The  Mean  Item  Value  associated  with  the 
set  of  "High  Item  Value"  items  became  the  "Go"  decision 
point  and  the  Mean  Item  Value  associated  with  the  set  of 
"Low  Item  Value"  items  became  the  "No  Go"  decision  point. 
Both  decision  point  values  are  depicted  in  Table  VIII. 


TABLE  VIII 


Purchase  Item  Ratings  and  Total /Mean  Item  Values 

I  ITEM  RATINGS 


CRITERIA 

WT. 

A1 

A2 

A3 

B1 

B2 

B3 

Design  Stability 

m 

10 

10 

10 

10 

9 

10 

Expected  Inv.  Life 

8.9 

10 

10 

9 

9 

8 

10 

Design  Complexity 

to 

• 

00 

3 

7 

9 

2 

2 

2 

Cost  of  Rep 1. Item 

7.4 

5 

3 

8 

5 

2 

4 

Antic.  Qty  of  Buy 

6.7 

9 

5 

4 

8 

5 

9 

Cost  of  Acq.  Data 

6.  1 

7 

6 

8 

1 

1 

1 

Amt  of  Crit.  Matls. 

5.  1 

10 

7 

8 

3 

7 

6 

Orgn/Reg.  Reqts 

4.4 

5 

5 

8 

3 

3 

2 

Ti me/Per s.  Reqts 

3.7 

4 

9 

10 

lO 

4 

2 

Sup.  Bus.  Solv. 

2.5 

1 

3 

10 

1 

1 

1 

Amt.  Fixed  Tooling 

1.8 

10 

8 

5 

8 

7 

4 

TOTAL  ITEM  VALUES 

457.2 

447.3 

531.6 

370.8 

306.5 

353. 

MEAN  ITEM  VALUES 


478.7 


343.5 


Step  48.  In  order  to  infer  that  there  existed  a  true 
difference  between  the  decision  points,  a  statistical  test 
for  the  difference  between  two  means  was  performed.  The 
test  performed  was  the  "Smal 1 -Sample  Inferences  about  the 
Difference  Between  Two  Population  Means:  Independent 
Sampling"  (See  Appendix  D) .  The  following  computations  are 
the  results  of  the  test  for  a  difference  between  two  popula¬ 


tion  means: 


H0  s  The  mean  of  the  “High  Item  Value"  items  equals  the 
mean  o-f  the  “Low  Item  Value"  Items. 

:  The  two  means  are  significantly  different. 

n  =  The  number  of  items  in  each  population  C3D. 
xt  *s  The  "High  Item  Value"  mean  C 478. 73. 

Xj =  The  "Low  Item  Value"  mean  C343.53. 

st =  Std  deviation  of  the  "High  Item  Values"  C 46. 0793. 

st=  Std  deviation  of  the  "Low  Item  Values"  C33.2153. 


Test  *  t’  value  (based  on  a  *  .05  significance  level  *  2.920 
and  (n-1)  degrees  of  freedom) 

Since  the  calculated  ’t’  value  exceeded  the  test  value 
the  null  hypothesis  could  be  rejected.  This  indicated  that 
there  was  a  significant  difference  between  the  "So"  and  the 
"No  Go"  decision  points.  This  roughly  means  that  the  model 
allows  for  a  meaningful  distinction  between  a  decision  to 
purchase  acquisition  data  and  a  decision  to  not  purchase 
acquisition  data. 


Research  Question  #2  Answered 


SECTION  I 


CRITERIA 


1)  Design  Stability 

2)  Expected  Inventory  Life 

3)  Complexity  of  Design 

4)  Cost  of  the  Replenishment  Item 

5)  Anticipated  Quantity  of  Buy 

6)  Cost  of  the  Acquisition  Data 

7)  Amount  of  Critical  Materials 

8)  Organizational /Regulatory  Requirements 

9)  Time/Personnel  Requirements 

10)  Original  Supplier’s  Business  Solvency 

11)  Amount  of  Fixed  Tooling 
TOTAL  ITEM  VALUE  (TIV) 


WT  X  RT  = 


If  TIV  >=  478.7  . Purchase  Acquisition  Data 

If  TIV  <  =  343.5  ....  DO  NOT  Purchase  Acquisition  Data 


If  343.5  O  TIV  <=  478.7 


YOUR  CHOICE 


Figure  9.  Decision-Making  Model 


Research  Phase  III 

Research  Phase  III  was  designed  to  answer  Research 
Question  # 3  which  stated:  Does  the  model  accurately  deter 
mine  when  acquisition  data  should  be  procured  and  when  it 


should  not  be  procured? 


Step  #10.  To  test  the  model,  nine  purchase  items  were 
randomly  selected  by  the  ASD  Directorate  of  Contract  Policy 
and  Review.  The  decision  to  purchase  acquisition  data  had 
already  been  made  for  the  items.  Using  the  developed  model, 
each  purchase  item  was  analyzed  by  one  item  manager  who  was 
familiar  with  neither  the  model  nor  the  purchase  item.  The 
results  of  the  Item  Manager's  analysis  of  the  purchase  items 
are  shown  in  Table  IX.  In  Table  IX,  the  symbol  repre¬ 

sents  the  final  ranking  of  each  criterion  as  shown  in  Table 
VI.  In  addition,  "TIV"  stands  for  "Total  Item  Value." 

The  nine  purchase  items  that  were  selected  for  analysis 
and  a  comparison  of  the  actual  purchase  decisions  with  the 
model’s  recommended  decisions  are  shown  in  Table  X. 

TABLE  IX 

Ratings  and  Total  Item  Values  of  Purchase  Items 


ITEM  RATINGS 


8 

9 

10 

9 

7 

5 

7 

9 

7 

8 

8 

7 

7 

10 

8 

8 

6 

2 

10 

9 

7 

10 

10 

5 

7 

7 

8 

3 

4 

W 

9 

2 

8 

1 

7 

9 

6 

4 

3 

7 

1 

4 

4 

1 

7 

4 

6 

5 

1 

9 

7 

7 

5 

5 

5 

3 

2 

2 

7 

10 

8 

5 

7 

2 

10 

9 

8 

8 

9 

6 

5 

8 

7 

9 

8 

4 

7 

2 

3 

6 

5 

6 

5 

5 

TABLE  X 


Purchase  Items  and  Test 

PURCHASE  ITEMS 

Results 

ACTUAL 

DECISION 

MODEL 

DECISION 

Al.  CW  65  Anti -Exposure  Suit 

BUY 

BUY 

A2.  CW  a 4  Cold  Weather  Flight  Cover 

BUY 

BUY 

A3.  HBU-12/A  Aircraft  Lap  Belt 

BUY 

CHOICE 

A4.  HU-55  Aircrew  Flight  Helmet 

BUY 

BUY 

A5.  EW  Jamming  Assignment  Computer 

DON’T  BUY 

DON’T  BUY 

A6.  Information— Digitizing  Encoder 

BUY 

CHOICE 

A7.  Transmitter — Directing  Exciter 

DON’T  BUY 

DON’T  BUY 

A8.  Digital  Radio  Frequency  Memory 

DON’T  BUY 

DON’T  BUY 

A9.  EF— 111  Interface  Adapters  (ATE) 

DON’T  BUY 

DON’T  BUY 

Step  #1 1 .  Conclusions  about  the  decision-making  model 
and  recommendations  -for  further  testing  will  be  outlined  in 
Chapter  VI. 

Research  Quest i on  #3  Answered 

In  testing,  the  model  successfully  predicted  a  decision 
to  purchase  acquisition  data  for  seven  of  nine  (77. 8%)  put — 
chase  items.  For  the  other  two  purchase  items,  the  model’s 
results  were  not  conclusive.  For  this  randomly  selected 
sample  of  purchase  items,  automatic  use  of  the  model  would 
have  duplicated  the  actual  decision  in  the  seven  cases  where 
results  were  conclusive. 


Conclusion 


This  chapter  has  taken  the  reader  through  a  step-by- 
step  problem  solving  and  data  analysis  approach  to  answer 
the  three  Research  Questions  posed  by  the  researchers.  In 
the  end,  the  researchers  developed  and  tested  a  heuristic 
acquisition  data  decision-making  model.  The  criteria  and 
decision  points  contained  in  the  model  will  be  discussed  in 
greater  detail  in  the  -following  chapter. 


V.  The  Decision-Making  Model 


Introduction 

This  chapter  introduces  the  acquisition  data  decision¬ 
making  model  developed  and  tested  by  the  researchers  (See 
Figure  10  for  a  copy  of  the  final  decision-making  model).  A 
brief  explanation  of  each  of  the  eleven  decision  criteria 
Mill  be  presented  along  with  a  description  of  the  rating 
scale  for  each  criterion  and  how  the  model  is  intended  to  be 
used. 

The  Cri teria 

The  acquisition  data  decision-making  model  is  comprised 
of  eleven  general  criteria  in  descending  order  of  impor — 
tance.  Each  criteria  has  an  associated  weight  which 
reflects  the  relative  importance  of  that  criterion  in  the 
decision  to  purchase  or  not  purchase  aquisition  data. 

Design  Stabi 1 i ty.  There  are  numerous  descriptions  of 
what  constitutes  design  stability.  Perhaps  the  most  widely 
used  definition  for  a  stable  item  is:  an  item  which  is 
baselined  for  production.  In  other  words,  an  item  which 
has  a  design  ready  to  move  into  production  is  a  stable  item. 
Another  definition  would  be:  an  item  which  has  had  few 
engineering  changes  to  its  configuration. 

Expected  Inventory  Life.  Expected  Inventory  Life  is 
the  length  of  time  the  item  is  expected  to  be  used  by  the 


military  service,  not  the  expected  wear  life  of  the  item. 


ITEM  DESCRIPTION 


ORG’N 


SECTION  I 

CRITERIA 

WT 

<  RT 

=  AMT 

1) 

Design  Stability 
(Low  =  1;  High  =  10) 

9.5 

21 

Expected  Inventory  LiTe 
(Low  =  1;  High  =  10) 

8.9 

3) 

Complexity  oT  Design 
(High  «  1;  Low  *  i0) 

8.3 

4) 

Cost  oT  the  Replenishment  Item 
(Low  =»  1;  High  ■  10) 

7.4 

5) 

Anticipated  Quantity  oT  Buy 
(Low  »  1;  High  =  10) 

6.7 

6) 

Cost  oT  the  Acquisition  Data 
(High  =  1 ;  Low  *  10) 

6.  1 

7) 

Amount  oT  Critical  Materials 
(High  =1;  Low  *  10) 

5.  1 

8) 

Organ i zat i onal /Regul atory  Requi rements 
(No  Push  =•  lj  Strong  Push  =*  10) 

4.4 

9) 

Time/Personnel  Requirements 
(Many  =  1;  Few  =  10) 

3.7 

10) 

Original  Supplier’s  Business  Solvency 
(Good  »  1;  Poor  *  10) 

2.5 

11) 

Amount  oT  Fixed  Tooling 
(Great  =*  1;  Little  =  10) 

1.8 

TOTAL  ITEM  VALUE  (TIV) 

SECTION  II 

IT  TIV  >=  478.7  ........  Purchase  Acquisition  Data 


IT  TIV  <-  343.5  ....  DO  NOT  Purchase  Acquisition  Data 
IT  343.5  <=  TIV  O  478.7  . . YOUR  CHOICE  !?!  I 


Figure  10.  Final  Acquisition  Data  Decision-Making  Model 
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Complexity  o-f  Design.  Complexity  of  design  can  best  be 
defined  as  the  combined  outcome  of  the  interaction  of  the 
following  main  determinants: 

(1)  The  number  of  elements  comprising  the  item, 

(2)  The  attributes  of  the  specified  elements  of 
the  item, 

(3)  The  number  of  interactions  among  the  specified 
elements  of  the  item,  and 

(4)  The  degree  of  organization  inherent  in  the  item 
design  (31:79). 

Cost  of  the  Rep 1 en i shmen t  Item.  The  cost  of  the 
replenishment  item  is  the  total  cost  of  the  item  being 
purchased,  including  all  Life  Cycle  Costs.  * 

Anticipated  Quantity  of  Buy.  Anticipated  quantity  of 
buy  does  not  simply  reflect  the  quantity  of  buy  for  the 
immediate  contract  or  initial  spares.  It  reflects  the  total 
estimated  quantity  of  buy  for  this  particular  item  over  the 
entire  expected  inventory  life. 

Cost  of  the  Acquisition  Data.  The  cost  of  the  acquisi¬ 
tion  data  is  the  cost  to  the  Sovernment  for  purchasing  all 
the  necessary  data  required  to  build  an  exact  copy  of  the 
item  or  component.  In  other  words,  this  cost  would  include 
not  only  the  cost  of  the  acquisition  data  package,  but  those 
additional  costs  known  as  "offsetting  costs."  These  offset¬ 
ting  costs  might  be  purchasing  personnel  costs,  administra- 


tive  costs,  the  costs  of  acquiring  proprietary  or  unlimited 
rights,  or  the  costs  associated  with  ensuring  that  the  data 
package  is  complete  and  accurate. 

Amount  of  Critical  Materials.  The  amount  of  critical 
materials  includes  critical  materials  used  in  the  item  or 
component  itself,  and  those  used  in  the  manuf acturing 
process. 

Organizational /Regulatory  Requi r ement s .  Organization¬ 
al/regulatory  requirements  are  a  measure  of  the  degree  to 
which  higher  headquarters,  or  local  Government  or  base 
organizations  press  for  the  purchasing  of  acquisition  data. 

Time/Personnel  Requirements.  Time/personnel  require¬ 
ments  are  a  reflection  of  the  amount  of  personnel  dedicated 
to  the  acquisition  data  procurement  process.  This  criteria 
is  really  a  cost  effectiveness  issue.  Are  there  enough 
personnel  available  within  the  SPO  to  adequately  ensure  that 
the  acquisition  data  to  be  purchased  will  be  complete  and 
accurate?  Do  the  personnel  assigned  to  the  purchasing  and 
evaluation  of  acquisition  data  packages  have  enough  time  to 
adequately  review  those  packages? 

Or iginal  Sup pi i er ’ s  Business  Sol vency.  The  "Original 
Supplier’s  Business  Solvency"  criteria  is  an  assessment  as 
to  the  financial  and  business  strength  of  the  -firm  who 
originally  developed  the  item  or  component  to  be  purchased. 
Does  it  appear  that  this  firm  will  be  in  business  with  the 
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Department  of  Defense  for  a  long  time  to  come,  or  Mill  he 


no  longer  maintain  the  Facilities  to  produce  the  item  soon 
after  contract  performance  is  complete? 

Amount  of  Fixed  Tooling.  This  criterion  requires  an 
assessment  as  to  the  amount  of  fixed  tooling  needed  to 
produce  the  item  for  which  we  are  purchasing  acquisition 
data. 

Using  the  Acquisition  Data  Decision-Making  Model 

Rat i ng  the  Criteria.  The  first  step  in  using  the 
acquisition  data  decision-making  model  is  rating  each 
criterion.  The  user  must  rate  the  item  for  which  the 
Government  intends  to  buy  acquisition  data  against  each  of 
the  eleven  criteria.  The  rating  scale  for  each  criterion  is 
a  continuous  scale  from  one  to  ten  based  on  the  degree  to 
which  the  acquisition  item  satisfies  each  criterion.  Pro¬ 
vided  below  are  characteristics  associated  with  the  two 
extremes  of  the  rating  scale  for  each  criterion.  Remember, 
each  criterion  can  be  rated  as  any  number  along  the  continu¬ 
um  of  one  to  ten. 

Design  Stabi 1 i ty .  If  the  item  has  a  very  low 
design  stability,  rate  it  as  a  one.  If  the  item  has  a  very 
high  degree  of  design  stability,  rate  it  as  a  ten. 

Expected  Inventory  Life.  If  the  item  being  pur — 
chased  has  a  low  expected  inventory  life,  rate  it  as  a  one; 
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if  it  has  a  very  high  expected  inventory  life,  rate  it  as  a 
ten.  Use  other  similar  items  as  a  basis  of  comparison  for 
determining  high  or  low  expected  inventory  life. 

Complexity  of  Design.  If  the  item  being  purchased 
has  a  very  high  design  complexity,  rate  it  as  a  one;  if  it 
is  very  non-complex,  rate  it  as  a  ten. 

Cost  of  the  Replenishment  Item.  Relative  to  other 
items  of  a  similar  nature,  if  this  item  has  a  very  low  life 
cycle  cost,  it  should  be  rated  as  a  one.  If  the  item  has  a 
relatively  high  life  cycle  cost,  rate  it  as  a  ten. 

Amount  of  Critical  Materials.  If  the  item  being 
purchased  has  a  very  high  content  of  critical  materials,  or 
if  many  of  the  materials  are  very  difficult  to  obtain,  rate 
it  as  a  one;  if  it  has  a  very  low  amount  of  critical 
materials,  rate  it  as  a  ten. 

Organisational /Regulatory  Reguirements.  If  there 
is  very  little  pressure  from  higher  levels  of  management  to 
purchase  acquisition  data  from  the  prime  contractor,  rate 
this  criterion  as  a  one.  If  higher  management  very  strongly 
recommends  purchasing  acquisition  data,  rate  this  criterion 
as  a  ten.  NOTE:  It  is  often  left  to  the  discretion  of  the 
Program  Manager  to  make  the  final  determination  as  to 
whether  or  not  the  data  is  necessary.  If  this  is  the  case, 
rate  this  criterion  as  a  one  if  the  Program  Manager  has  full 


authority  to  make  the  acquisition  data  procurement  decision. 


>v»v.\x  , 
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Ti me/Per sonnel  Requirements.  If  in  the  user’s 
opinion  purchasing  and  evaluating  the  acquisition  data 
requires  a  large  amount  of  personnel  and  time  to  thoroughly 
review  the  data  packages,  rate  this  criterion  as  a  one. 
Moreover,  i-f  the  time  and  personnel  costs  o-f  purchasing  the 
data  greatly  exceed  the  cost  o-f  the  data  itself,  rate  this 
criterion  as  a  one.  In  other  words,  if  time/personnel 


requirements  are  great,  rate  this  criterion  as  a  one.  If 
the  acquisition  and  evaluation  process  is  minimal  and 
requires  only  a  few  personnel  to  do  the  job,  rate  this 


criterion  as  a  ten. 


Original  Suppl ier  *  s  Business  Solvency.  If  it 
appears  that  the  original  supplier  of  the  item  for  which  the 

ft 

Government  wishes  to  purchase  acquisition  data  will  most 
likely  still  have  the  facilities  to  build  the  item,  rate 


this  criterion  as  a  one.  If  it  appears  that  the  original 
supplier’s  business  solvency  is  not  strong,  rate  this  item 
as  a  ten. 


Amount  of  Fixed  Tool inq.  If  the  item  for  which 
the  Government  intends  to  purchase  acquisition  data  requires 
a  very  large  amount  of  fixed  tooling  in  its  manufacturing 
process,  rate  this  criterion  as  a  one.  If  the  item  uses 


very  little  fixed  tooling  in  its  manuf acturing  process,  rate 


this  criterion  as  a  ten 


Calculating  the  Total  Item  Value  (TIV) .  Once  each 
criterion  is  rated  -from  one  to  ten,  the  user  should  multiply 
the  weight  -for  each  criterion  by  the  rate  just  assigned. 

The  product  is  then  placed  in  the  column  marked  “AMT."  The 
user  must  then  sum  all  the  products  to  arrive  at  a  TIV. 

Determining  the  Outcome.  The  final  step  in  using  the 
acquisition  data  deci sion -making  model  is  to  compare  the 
calculated  TIV  with  the  high  and  low  decision  points  listed 
under  Part  I I  of  the  model.  If  the  TIV  is  equal  to  or 
greater  than  478.7,  it  is  recommended  that  the  decision¬ 
maker  (user)  purchase  acquisition  data  for  that  item.  If 
the  TIV  is  equal  to  or  less  than  343.5,  it  is  recommended 
that  the  decision-maker  not  purchase  acquisition  data  for 
that  item.  If  the  TIV  falls  between  343.5  and  478.7,  it  is 
left  to  the  decision-maker’s  discretion  to  purchase  or  not 
purchase  acquisition  data  for  the  item  being  acquired. 


VI. 


Finding^  and  Recommendat ions 
■for  Future  Research 


Research  Summary 

This  research  e-f-fort  was  undertaken  to  provide  a  model 
which  will  help  determine  when  it  is  appropriate  to  purchase 
or  not  purchase  acquisition  data  -for  a  particular  item.  To 
accomplish  this  task,  the  researchers  looked  at  several 
popular  decision-making  techniques  and  chose  the  one  most 
appropriate  -for  solving  the  problem  set  forth  in  Chapter  I. 
The  researchers  chose  the  technique  known  as  the  Kepner- 
Tregoe  approach  -for  problem  solving  and  decision-making. 

Because  the  Kepner — Tregoe  approach  is  mostly  used  to 
help  select  one  alternative  among  many,  the  researchers  had 
to  modify  the  technique  so  that  it  would  be  useful  in 
situations  where  alternate  courses  of  action  are  not  always 
available.  This  meant  that  the  researchers  had  to  calculate 
two  decision  paints  —  a  "Go"  (Buy  acquisition  data)  and  a 
"No  Go"  (Do  not  buy  acquisition  data)  decision  point. 

The  researchers’  plan  of  attack  consisted  of  breaking 
the  effort  down  into  three  major  phases.  The  first  phase 
corresponded  with  the  researchers’  first  objective,  to 
identify  and  rank  order  the  factors  relevant  to  the  acquisi¬ 
tion  data  purchasing  decision.  The  second  phase  correspon¬ 
ded  with  the  second  research  objective,  to  use  the  factors 
from  objective  1  to  develop  a  decision-making  model.  Final¬ 
ly,  phase  three  corresponded  with  the  third  research  objec¬ 
tive,  to  test  the  model  which  was  developed. 


Research  Objective  #1 

Findings  for  Objective  #1.  The  researchers  used  a 
three  step  process  (See  Chapter  III,  Steps  #1  through  #3)  to 
achieve  their  first  objective.  Through  personal  interviews 
and  a  thorough  literature  review,  the  researchers  obtained 
eleven  criteria  most  often  used  in  the  acquisition  data 
decision-making  process.  These  criteria  were  used  in  the 
final  model  (See  Chapter  V,  Figure  10)  to  determine  when  it 
is  appropriate  to  purchase  acquisition  data  for  a  particular 
item. 


Recommendat ions  for  Objective  #1.  The  Phase  1  effort 
(identifying  and  ranking  criteria)  of  the  final  decision¬ 
making  model  was  quite  strong.  The  random  sampling  inter — 
view  techniques  used  by  the  researchers,  as  well  as  the 
efforts  of  several  previous  theses,  in  attempting  to  identi¬ 
fy  the  criteria  relevant  to  the  decision-making  process  were 
very  fruitful.  It  is  always  possible  to  find  "just  one 
more"  specific  criterion  if  you  interview  several  data  mana¬ 
gers,  but  the  criteria  identified  in  this  research  effort 
are  generic  enough  to  encompass  them. 

The  results  of  the  experts’  rankings  of  the  criteria 
were  somewhat  surprising.  The  researchers  did  not  antici¬ 
pate  such  a  strong  agreement  among  the  experts’  rankings 
because  both  Air  Force  Systems  Command  and  Air  Force  Logis¬ 
tics  Command  personnel  were  used  to  rank  the  items.  During 


their  preliminary  research  under  this  topic,  the  researchers 


found  a  distinct  difference  in  opinion  about  what  was  con¬ 
sidered  important  in  the  decision  to  purchase  acquisition 
data.  This  in  itself  was  not  so  startling  due  to  the  dif¬ 
ferences  in  roles  played  by  each  command.  Air  Force  Systems 
Command  is  concerned  with  an  item  before  it  moves  into  the 
operational  Air  Force,  while  Air  Force  Logistics  Command  is 
mainly  concerned  with  supporting  the  item  after  it  is  placed 
in  operation. 

Nonetheless,  most  of  the  experts  (80%)  ranked  "design 
stability"  either  first,  second  or  third  in  importance.  The 
next  five  criteria  —  expected  inventory  life,  complexity  of 
design,  cost  of  the  replenishment  item,  anticipated  quantity 
of  buy,  and  cost  of  the  acquisition  data  —  were  all  ranked 
fairly  close  in  importance.  The  last  five  criteria  — 
amount  of  critical  materials,  organizational /regulatory 
requirements,  time/personnel  requirements,  business  solven¬ 
cy,  and  amount  of  fixed  tooling  —  were  ranked  well  below 
the  first  six  criteria. 

It  might  be  worthwhile  for  an  analysis  to  be  performed 
to  determine  if  there  is  a  significant  difference  between 
the  opinions  of  Air  Force  Systems  Command  and  Air  Force 
Logistics  Command  personnel.  When  funding  for  a  program  is 
cut  back,  the  area  which  most  often  bears  the  brunt  of  the 
funding  reduction  is  logistics  support  (spares,  initial 
provisioning,  training  support,  etc.).  If  we  can  identify 
those  criteria  most  pressing  to  each  command,  perhaps 
program  managers  can  be  more  equitable  in  their  cutbacks. 


Research  Objective  #2 


Findings  -for  Ob jecti ve  #2.  The  researchers*  second 
objective  entailed  developing  a  model  to  assist  Program 
Managers  in  their  decisions  concerning  the  procurement  o-f 
acquisition  data.  Chapter  V  describes  in  detail  each  of  the 
criteria  found  under  Phase  1  of  this  research  effort,  and  it 
explains  how  to  use  the  model.  The  researchers  used  a  six 
step  process  (See  Chapter  III,  Steps  #4  through  #9)  to 
arrive  at  a  working  acquisition  data  decision-making  model. 

The  Kepner — Tregoe  approach  dictates  that  each  of  the 
criteria  found  under  Phase  1  be  weighted  on  a  scale  from  one 
to  ten.  Even  though  there  was  a  fairly  distinct  break 
between  the  first  six  and  the  Last  five  criteria  under  the 
experts’  rankings  of  those  items,  there  was  a  gradual  reduc¬ 
tion  in  the  weighting  of  all  the  criteria.  In  other  words, 
there  was  no  tremendous  difference  between  the  weight  of  one 
criterion  and  the  next  lower  criterion.  This  gives  credence 
to  the  criteria  as  a  whole  in  their  importance  or  usefulness 
in  determining  if  acquisition  data  is  required. 

In  addition  to  weighting  the  criteria,  the  researchers 
had  to  develop  a  "Go"  and  "No  Go"  decision  point  for  the 
model.  To  derive  the  "Go"  decision  point,  item  managers 
rated  items  for  which  acquisition  data  was  purchased  and 
used  under  a  competitive  reprocurement .  The  "No  Go"  point 
was  derived  by  item  managers  rating  items  for  which  no  data 
was  purchased.  The  researchers  found  that  it  was  mostly  on 


sub— assembl i es  or  larger  components  that  the  government 
chose  not  to  purchase  acquisition  data,  while  it  was  smal¬ 
ler,  less  complex  items  that  the  government  purchased  the 
acquisition  data. 

Using  the  test  for  the  difference  between  two  means 
described  in  Appendix  0,  the  researchers  found  that  there 
indeed  existed  a  significant  difference  between  the  "Go"  and 
"No  Go"  decision  points.  Had  this  test  proved  inconclusive, 
it  would  have  been  meaningless  for  a  Program  Manager  to  use 
the  model  for  deciding  whether  or  not  to  purchase  acquisi¬ 
tion  data.  By  showing  that  there  exists  a  true  difference 
between  the  two  decision  points,  the  second  objective  is 

fulfilled  —  the  researchers  have  now  arrived  at  a  poten- 
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tially  useful  model. 

Recommendat ions  for  Qb  jecti ve  #2.  There  are  several 
areas  of  the  acquisition  data  decision-making  model  which 
would  benefit  from  further  testing.  First  of  all,  due  to 
time  constraints,  the  researchers  were  only  able  to  select 
three  items  to  develop  the  "Go"  decision  point,  and  three 
items  to  develop  the  "No  Go"  decision  point.  Since  the  two 
decision  points  are  crucial  to  the  model’s  success,  it  would 
be  very  beneficial  to  select  a  larger  sample  size  of  items 
for  which  acquisition  data  was  and  was  not  purchased. 

In  addition,  it  might  be  interesting  to  study  the 
effects  on  the  decision  points  if  similar  types  of  items  are 
selected  for  both  the  "Go"  and  "No  Go"  points.  There  may 


also  be  a  correlation  between  the  type  of  item  selected  -for 
evaluation  and  the  value  of  the  decision  point.  In  other 
words,  are  there  different  decision  points  for  different 
Program  Offices? 

Another  area  for  further  study  might  be  the  concept  of 
having  only  one  decision  point.  If  the  Total  Item  Value 
falls  above  this  point,  buy  acquisition  data.  If  the  Total 
Item  Value  falls  below  this  point,  do  not  buy  the  data.  The 
researcher  should  investigate  the  potential  lack  of  flexi¬ 
bility  that  this  approach  might  present  to  the  decision¬ 
maker  . 

Research  Objective  #3 

Findings  for  Objective  #3.  The  third  objective  of  the 
researchers  was  to  test  the  validity  of  the  model.  The 
researchers’  approach  was  to  randomly  select  purchase  items 
for  which  a  decision  has  already  been  made  regarding  the 
purchase  of  acquisition  data.  The  researchers  then  compared 
the  model's  recommended  decision  with  the  actual  decision 
made  by  Program  Managers.  This  objective  encompassed  Steps 
#10  and  #11  of  the  research  methodology  presented  in  Chapter 
III. 

The  researchers  selected  nine  items,  and  the  model 
accurately  predicted  seven  of  the  nine  items.  The  remaining 
two  decisions  proved  inconclusive,  for  the  model  recommended 
the  choice  be  left  to  the  deci si on-maker . 


Recommendat i ons  for  Objective  #3.  Before  any  new  model 
can  be  used  in  field  operations,  it  is  essential  that  the 
model  be  thoroughly  tested.  Consequently,  this  is  the  area 
of  the  researchers’  study  that  requires  support.  The  random 
sample  of  purchase  items  used  to  test  the  model  was  somewhat 
limited,  again  due  to  time  constraints.  Therefore,  the 
researchers  recommend  that  the  bulk  of  any  follow-on  work  be 
spent  on  testing  the  acquisition  data  decision-making  model. 

It  needs  to  be  determined  if  the  model  holds  up  under 
the  test  of  purchase  items  of  various  sizes,  complexities, 
design  and  compositions.  It  also  needs  to  be  determined  if 
the  model  gives  similar  recommendations  to  various  types  of 
item,  or  if  the  model’s  accuracy  diminishes.  Finally,  it 
would  be  interesting  to  determine  if  the  model  is  consistent 
in  its  recommendations  for  items  within  the  same  program,  or 
even  within  the  same  Program  Office. 

In  order  for  the  model  to  be  useful  as  a  decision¬ 
making  tool,  it  must  stand  the  test  of  all  possible  combina¬ 
tions  of  acquisition  conditions,  program  types,  and  purchase 
item  types.  Although  the  researchers  accomplished  their 
final  objective  of  testing  the  validity  of  the  decision¬ 
making  model,  the  strength  of  the  model  depends  upon  the 
outcome  of  future  tests. 

General  Observations  About  the  Model 

In  many  respects,  the  acquisition  data  decision-making 
model  developed  by  the  researchers  is  rather  unique.  Unlike 


many  other  Operations  Research  modeling  or  decision-making 
techniques,  the  Kepner-Tregoe  approach  does  not  require 
quantif ication  of  the  variables  under  study.  Just  glancing 
at  the  criteria  gives  the  user  an  idea  that  there  are  many 
subjective  aspects  to  this  model.  For  example,  how  does  one 
accurately  define  "design  stability,"  or  "item  complexity," 
or  the  “original  supplier’s  business  solvency?"  Ask  ten 
individuals  about  the  same  item  and  you  are  likely  to  get 
ten  different  opinions. 

But  that  is  the  advantage  of  this  Kepner — Tregoe 
approach.  Because  each  of  the  criteria  are  weighted,  they 
absorb  the  subjectivity  associated  with  an  opinionated 

rating  system.  The  weight  associated  with  each  criterion  is 

% 

composed  of  other  experts’  opinions  .  .  .  the  weight  is 
simply  an  average  of  opinions  from  experts  with  twenty  years 
service  to  experts  with  four  years  service. 

Therefore,  it  does  not  matter  if  the  decision-maker  has 
a  limited  familiarity  with  the  acquisition  data  environment. 
If  the  decision-maker  knows  enough  about  the  particular  item 
he  is  purchasing  (we  would  assume  so,  or  else  he  should  not 
be  in  the  position  to  make  the  decision  whether  or  not 
acquisition  data  should  be  purchased)  to  have  a  feel  for  its 
design  stability,  item  complexity,  and  so  on,  a  well-tested 
model  should  give  strong  guidance  for  a  good  decision. 

In  Chapter  II,  the  researchers  discussed  the  acquisi¬ 
tion  data  "time"  problem.  Briefly  stated,  this  time  problem 
is  a  situation  where  defining  acquisition  data  requirements 
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at  the  onset  of  a  program  represents  the  lowest  cost  of  data 
to  the  Government.  However,  data  experts  state  that  the 
earliest  -feasible  time  to  define  acquisition  data  require¬ 
ments  is  not  until  after  the  Physical  Configuration  Audit. 

By  eliminating  some  of  the  uncertainty  surrounding  the 
need  for  acquisition  data,  the  decision-making  tool  proposed 
in  this  thesis  has  the  potential  to  significantly  reduce 
the  cost  of  that  data.  Using  the  hypothesized  model  can 
help  reduce  costs,  for  it  <1)  allows  the  decision— maker  to 
purchase  the  data  early  in  the  contracting  process,  and 
thereby  negotiate  reasonable  data  costs  when  the  Government 
ha?  a  stronger  bargaining  position,  and  (2>  allows  the 
decision-maker  to  decide  not  to  purchase  acquisition  data 
when  it  is  inappropriate  to  do  so. 

As  mentioned  above,  the  researchers  feel  that  further 
testing  would  be  beneficial  before  one  attempts  to  opera¬ 
tionalize  this  model.  However,  if  the  model  fails  this 
testing,  the  results  of  this  research  effort  are  still 
significant,  for  we  now  have  a  complete  listing  of  criteria 
important  to  the  acquisition  data  decision-making  process. 
That  in  itself  is  a  major  stride  forward  to  making  a  better 
decision.  Having  eleven  criteria  instead  of  just  the  two  or 
three  presently  used,  brings  program  and  data  managers 
closer  to  making  a  decision  under  certainty  rather  than 
making  a  decision  under  uncertain  conditions. 

One  final  observation  which  comes  from  the  researchers’ 
contacts  with  the  various  "experts"  on  acquisition  data. 


The  decision-maker  will  be  able  to  justify  purchasing  acqui¬ 
sition  data,  if  he  so  chooses,  regardless  of  the  outcome  of 


the  acquisition  data  decision-making  model.  Although  more 
difficult  to  do  so  with  the  current  emphasis  on  competitive 
acquisition,  the  deci sion-maker  will  also  be  able  to  justify 
not  purchasing  acquisition  data,  if  he  so  chooses. 

As  with  any  other  decision-making  model,  this  model  is 
simply  a  tool  to  be  used  when  the  decision-making  process 
becomes  too  complex.  It  is  not  a  panacea;  nor  is  it 
supposed  to  be  a  panacea.  In  the  final  analysis,  the 
decision-maker  must  make  the  decision,  not  the  decision¬ 


making  model 


Appendix  A:  The  Delphi  Technique 


Background 

The  Delphi  is  an  "organized  brainstorming  and  consensus 

technique  which  can  be  used  Tor  many  problems  which  defy 

quantification  or  preciseness  C8:1913." 

Essentially,  the  del phi  is  a  series  of  intensive 
interrogations  of  each  individual  expert  (by  a 
series  of  questionnaires)  concerning  some  primary 
question  interspersed  with  controlled  feedback. 

The  procedures  are  designed  to  avoid  direct 
confrontation  of  the  experts  with  one  another 
C27: 381-3883. 

The  developers  of  the  Delphi  technique  (RAND  Carp) 
argue  that  the  procedures  are  more  conducive  to  independent 
thought  and  "allow  more  gradual  formulation  to  a  considered 
opinion  C27: 381-3823 . " 

Trying  to  establish  a  consensus  from  within  a  group  can 
lead  to  some  very  significant  problems.  First  of  all,  the 
group  consensus  is  "highly  influenced,  if  not  determined,  by 
the  views  of  the  member  of  the  group  who  does  the  most 
talking  .  .  .  C9:33."  A  second  problem  is  "noise  —  irrele¬ 
vant  or  redundant  material  that  obscures  the  directly  rele¬ 
vant  material  offered  by  participants  C9:33."  A  final 
significant  problem  is  the  group  pressure  placed  on  indi¬ 
viduals  to  follow  the  group’s  consensus,  whether  it  be 
active  or  subconscious  pressure  (9:3). 


The  Delphi  technique  was  developed  to  counter  these 
three  problem  areas.  The  Delphi  method  gathers  information 
from  and  deals  "separately  with  experts  who  have  generally 


the  same  specialized  capabilities  C8sl913."  All  opinions 
are  recorded  separately  from  one  another  and  then  a  summary 
of  the  results  is  passed  to  the  group,  thereby  diminishing 
the  first  problem  (9:3).  It  also  reduces  noise  by  using 
controlled  feedback  which  “summarizes  relevant  opinions 
given  on  specific  items  within  the  survey  C28:273."  And 
finally,  the  Delphi  method  reduces  “group  pressure"  because 
results  are  expressed  in  “terms  of  a  statistical  score." 
Thus,  there  is  no  pressure  to  arrive  at  a  consensus  (10:4). 

Procedure 

The  first  task  under  the  Delphi  Technique  is  selecting 
the  "experts."  The  technique  "strives  to  get  the  Cexpertsl 
to  feel  free  to  give  their  best  inputs  by  submitting  them 
anonymously,  frankly  and  unfettered  by  face-to-face 
meetings  £8:191-1923."  After  they  are  chosen,  the  first  of 
what  may  be  several  questionnaires  is  issued.  The  indi¬ 
viduals  are  to  record  on  the  survey  instrument  their 
responses  regarding  a  particular  set  of  questions.  These 
responses  are  then  summarized  and  sent  back  to  the  respon¬ 
dees  along  with  a  mean  response  to  the  item  and  the  inter — 
quartile  range  of  responses. 

The  respondent  is  then  asked  to  reevaluate  his  original 
response  now  that  he  has  seen  the  feedback  from  all  respon¬ 
dents.  If  the  second  answer  lies  outside  the  interquarti le 
range,  he  is  asked  to  explain  his  reason  for  selecting  that 
response.  Again,  all  answers  are  tabulated  and  returned  to 
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the  respondees  with  a  summary  of  the  reasons  why  the  answers 
outside  the  interquarti le  range  were  chosen.  A  third  round 
of  interviews  is  performed,  thus  leading  us  to  a  convergence 
of  answers  among  the  experts  (10:7). 


Appendix  Bs  The  Kepner — Tregoe  Approach  to  Deci si on-Making 


Background 

The  Kepner — Tregoe  approach  to  decision-making  is 
founded  on  the  scientific  method  of  problem-solving.  The 
authors  of  this  heuristic  approach  to  decision-making  state 
that  while 

the  scientist  and  the  doctor  go  through 
elaborate  courses  of  training  in  order  to  learn 
objective  methods  of  using  information  in  the 
solution  of  technical  problems,  .  .  .  the  manager, 
however,  who  must  constantly  analyze  problems  and 
make  decisions,  and  who,  therefore,  perhaps  more 
than  anyone  else  needs  efficient  ways  of  working 
with  information,  has  been  left  to  shift  for 
himself.  The  result  is  usually  ...  a  hit-and- 
miss  approach  based  on  a  mixture  of  experience  and 
guesswor  k  C 25 : 40 1 . 

The  Kepner-Tregoe  approach  is  an  "orderly  system  for 
processing  information,  a  system  in  which  certain  steps 
follow  others  in  a  fixed  order  C25:403."  It  is  used  to 
compare  several  different  alternative  courses  of  action. 

The  authors  identify  seven  basic  concepts  of  decision¬ 
making: 

1.  Establish  decision  objectives 

2.  Classify  the  decisions  as  to  importance 

3.  Develop  alternative  actions 

4.  Evaluate  alternatives  against  the  established 
objectives 

5.  Select  tentative  decisions  (the  alternative  which 


best  achieves  all  objectives) 

6.  Explore  the  tentative  decision  for  possible  future 
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adverse  consequences 

7.  Control  the.  effects  of  the  final  decision  by  taking 
other  actions  to  prevent  possible  adverse  consequences  from 
becoming  problems,  and  make  sure  the  actions  decided  on  are 
carried  out  (25:46-50). 

Procedure 

The  procedure  for  accomplishing  the  Kepner — Tregoe 
approach  follows  very  closely  to  the  seven  basic  concepts 
identified  above.  First  of  all,  the  manager  must  identify 
the  objectives  which  he  intends  to  accomplish.  It  could  be 
an  objective  which  satisfies  an  organizational  goal,  or  it 
could  be  one  which  will  solve  a  known  problem  (25:48). 

Once  all  objectives  are  listed,  the  manager  determines 
which  objectives  are  "MUSTS"  (a  requirement  which  cannot  be 
compromised)  and  which  are  "WANTS"  (objectives  where  the 
manager  gets  the  best  possible  performance  from  a  decision  - 
subject  to  bargaining) .  The  "WANTS"  are  then  rank  ordered 
and  weighted  from  1  to  10  with  10  being  the  highest  (25:48). 

The  manager  then  decides  alternate  ways  of  getting  the 
job  done,  and  the  alternatives  are  evaluated  against  the 
"MUSTS"  and  "WANTS."  If  an  alternative  does  not  satisfy  all 
"MUSTS,"  it  is  discarded.  Each  alternative  is  rated  from  1 
to  10  against  each  "WANT"  objective,  and  the  rating  is 
multiplied  by  the  weighting  for  that  objective.  The  amounts 
for  each  objective  are  then  summed  to  obtain  a  total  score 
for  each  alternative  (25:48-49:  25:196-199). 


One  or  two  alternatives  with  the  highest  scores  are 
selected  as  tentative  decisions,  and  the  manager  attempts  to 
■find  adverse  consequences  with  each.  The  adverse 
consequences  are  given  probabilities  o-f  occurring  (rated  from 
1  to  10,  with  10  having  the  greatest  probability  of 
occurrence) ,  and  they  are  also  rated  from  1  to  10  as  to  the 
impact  of  the  consequence  should  it  actually  occur.  The 
rating  is  multiplied  by  the  probability,  and  all  values  are 
summed  for  each  alternative.  This  amount  is  then  subtracted 
from  the  score  obtained  during  the  selection  process,  and 
the  alternative  with  the  greatest  overall  value  is  the 
decision  which  best  serves  the  manager’s  interest  (25:49-50; 
25:201-205) . 

It  is  important  to  note  that  as  in  any  decision-making 
technique,  the  final  decision  to  go  or  not  go  with  a 
particular  alternative  rests  with  the  manager,  regardless  of 
the  value  obtained  from  the  model. 


Appendix  C:  The  Kendal  1  Coefficient  of  Concordances  W 
Background 

The  Kendall  Coefficient  of  Concordance  was  developed  as 
a  technique  to  ascertain  the  “overall  agreement  among  k  sets 
of  rankings  C33s2293."  This  measure  has  been  -found  quite 
useful  in  studies  of  inter test  or  inter judge  reliability. 

The  coefficient  of  concordance  is  “an  index  of  the  diver — 
gence  of  the  actual  agreement  shown  in  the  data  from  the 
maximum  possible  (perfect)  agreement  [33:2301.“ 

Procedure 

The  first  step  in  performing  the  Kendall  Coefficient  of 
Concordance  is  to  set  up  a  table  with  the  various  judges 
(“experts"  in  our  thesis)  along  the  left-hand  column  and  the 
items  to  be  ranked  (criteria  in  our  thesis)  along  the  top 
row.  The  table  is  then  filled  with  the  rankings  of  each 
judge  for  each  criterion  (See  Table  XI  for  an  example). 

TABLE  XI 

Ranks  Assigned  to  Acquisition  Data  Decision 
Criteria  by  Ten  "Experts" 


Cri teri a 


a 

b 

c 

d 

e 

f 

g 

h 

Expert  #1 

1 

4 

8 

6 

2 

3 

7 

5 

Expert  #2 

■ 

5 

7 

6 

1 

4 

2 

■ 

8 

• 

• 

Expert  #10 

n 

4 

8 

7 

1 

3 

■ 

■ 

5 

6 

SUMS 

21 

43 

75 

62 

15 

3<S 

41 

69 

As  you  can  see  from  this  example,  the  degree  of 
agreement  among  the  ten  experts  is  reflected  by  the  degree 


of  variance  among  the  eight  sums  of  ranks.  "W,  the  coef¬ 
ficient  of  concordance,  is  a  function  of  that  degree  of 
variance  C 33: 230-231 3 . " 

The  following  equation  is  used  to  calculate  W: 


W  =  _ s _  (5 

(1/12)  (kxk)  <IMS  -  N) 

where:  k  =*  the  number  of  "experts"  interviewed 

N  =  the  number  of  criteria  to  be  ranked 

s  =  sum  of  the  squares  of  the  observed  deviations 
from  the  mean  of  the  SUMS 


2 


In  the  example  above,  s  =*  (21 

-  45.25)  + 

(43  - 

45. 

2 

2 

2 

+  (75  -  45.25) 

+  (62  -  45.25) 

+  (15  -  45.25)  + 

2 

2 

2 

(36  -  45.25)  + 

(41  -  45.25)  + 

(69  -  45.25) 

=  3341.5 

W  - 

3341.5 

. 796  (  O 

<  W  < 

0 

(1/12) (10x10) (512  -  8) 


A  high  or  significant  value  of  W  implies  that  the 
experts  are  applying  the  same  standard  in  ranking  the 
criteria  under  study  (33:237). 

When  N  is  greater  than  seven,  the  expression  given  in 
the  following  formula  is  approximately  distributed  as  Chi 
Square  with  N  -  1  degrees  freedom. 

Chi-Square  «  k (N  -  1)W  (6) 


If  the  value  of  Chi-Square  as  computed  from  Equation 
equals  or  exceeds  that  shown  in  the  Chi  Square  table  in 
Appendix  E  for  a  particular  level  of  significance  and  a 
particular  value  of  df  C73,  then  the  "null  hypothesis  that 
the  k  C 103  rankings  are  unrelated  may  be  rejected  at  that 


level  of  significance  C33:2363." 


f 
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'  Appendix  Ds  Seal  1  -Sample  Inferences  about  the  Pi f f erence 

'  Between  Two  Population  Means: 

^  Independent  Sampl inq 

Background 

Often  times  you  want  to  make  inferences  about  the 
!  differences  between  two  populations'  means.  In  order  to  use 

this  test,  you  must  make  the  following  assumptions: 


1.  Both  sampled  populations  have  relative  frequency 
distributions  that  are  approximately  normal ; 


2.  The  population  variances  are  equal; 

3.  The  samples  are  randomly  and  independently  selected 
from  the  populations. 

Procedure 


STEP  #1; 


STEP  #2: 


Calculate  the  mean  for  samples  1  (high  item 
value)  and  sample  2  (low  item  value),  the  stan 
dard  deviations  for  samples  1  and  2,  and  the 
pooled  estimator,  Sp ,  using  the  following 
equat i on : 


s2  = 

(n 

>  1 

-  l)s2  +  (n  -  l)s2 

1  2  2 

(2) 

p 

Calculate 

the  t- 

n  +  n  -  2 

1  2 

-statistic  using  the  following 

equation: 

t  ». 

O 

i 

CN 

lx 

l 

lx 

(3) 

J  %’(*  *  *) 

1  2 

STEP  #3:  If  the  test  statistic  is  greater  than  the  t 

value  in  Appendix  E  at  (n  +  n  -  2)  df  and  an 
appropriate  level  of  significance,  reject  the  Null 
Hypothesis  that  there  is  no  difference  between  the 
two  decision  points  (7:340). 
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Appendix  Es  Abbreviated  Chi-Square  and  Student's  t-Tables 


TABLE  XII 
Chi-Square  Table 


LEVEL  OF  SIGNIFICANCE 
DEGREES  OF  - 


FREEDOM 

.  100 

.050 

.025 

.010 

.005 

5 

9.23635 

1 1 . 0705 

12.8325 

15.0863 

16.7496 

6 

10.6446 

12.5916 

14.4494 

16.8119 

18.5476 

7 

12.0170 

14.0671 

16.0128 

18.4753 

20. 2777 

8 

13.3616 

15.5073 

17.5346 

20. 0902 

21.9550 

9 

14.6837 

16.9190 

19.0228 

21.6660 

23.5893 

io 

15.9871 

18.3070 

20.4831 

23. 2093 

25. 1882 

11 

17.2750 

19.6751 

21.9200 

24.7250 

26.7569 

12 

18.5494 

21.0261 

23.3367 

26.2170 

28.2995 

13 

19.8119 

22.3621 

24.7356 

27.6883 

29.8194 

14 

21.0642 

23.6848 

26.1190 

29. 1413 

31.3193 

Source: 

(7:899) 

TABLE 

XIII 

Student ’ s 

t-Table 

LEVEL  OF  SIGNIFICANCE 

DEGREES  OF 

FREEDOM 

.  100 

.050 

.025 

.010 

.005 

1 

3.078 

6.314 

12.706 

31.821 

63.657 

2 

1.886 

2.920 

4.303 

6.965 

9.925 

0* 

1.638 

2.353 

3.  182 

4.541 

5.841 

4 

1.533 

2.  132 

2.776 

3.747 

4.604 

5 

1.476 

2.015 

2.571 

3.365 

4.032 

6 

1.440 

1.943 

2.447 

3.  143 

3.707 

7 

1.415 

1.895 

2.365 

2.998 

3.499 

8 

1.397 

1.860 

2.306 

2.896 

3.355 

9 

1.383 

1.833 

2.262 

2.821 

3.250 

10 

1.372 

1.812 

2.228 

2.764 

3.  169 

Source:  (7:887) 


Appendix  F:  Structured  Interview  Questionnaires 


Survey  #1 
INSTRUCTIONS: 

The  following  questionnaire  contains  a  list  of  criteria 
used  to  determine  if  acquisition  data  should  be  purchased 
for  a  particular  item.  We  ask  that  you  circle  the  number 
that  most  closely  corresponds  to  your  opinion  as  to  whether 
or  not  the  criterion  is  important.  If  you  feel  that  the 
criterion  is  of  little  importance  in  determining  whether  or 
not  acquisition  data  should  be  purchased,  circle  a  zero.  If 
the  criterion  is  somewhat  desirable,  circle  a  one,  and  so 
on.  If  you  rate  a  criterion  as  a  zero  or  a  five,  please 
comment  as  to  why  you  chose  this  rating.  Question  #12,  Part 
II,  allows  you  to  add  any  further  criteria  you  feel  may  be 
appropriate. 

We  also  ask  that  you  determine  if  a  criterion  should  be 
a  “MUST"  or  a  "WANT".  A  "MUST"  criterion  is  one  which,  in 
your  opinion,  absolutely  must  be  present  before  acquisition 
data  is  to  be  purchased.  If  this  criterion  does  not  exist, 
the  Government  may  not  purchase  the  acquisition  data.  A 
"WANT"  criterion  is  one  whifch  is  desirable,  but  it  is  not 
absolutely  necessary.  Please  check  the  appropriate  box 
below  each  criterion.  In  addition,  certain  questions 
will  ask  for  your  opinion  as  to  a  criterion's  definition. 
Please  be  as  succinct  as  possible  in  your  response. 

Finally,  Part  III  requests  you  to  rank  each  criterion 
from  most  important  to  least  important  (A  rank  of  "1"  is 
most  important).  When  ranking  the  criteria,  you  may  have 
ties  between  items.  If  so,  continue  with  the  rank  the 
next  criteria  would  have  had  if  there  were  no  ties.  For 
example,  if  you  had  to  rank  the  five  items  listed  below  and 
you  felt  that  two  were  equally  important,  your  ranking  might 
appear  as  follows: 


Criterion  Rank 


1.  Design  stability  5 

2.  Cost  of  the  data  2 

3.  Complexity  of  design  1 

4.  Regulatory  requirements  2 

5.  Expected  inventory  life  4 


Include  in  the  ranking  any  item  that  you  added  in  question  # 
12  under  Part  II.  You  may  use  the  blank  spaces  to  add  these 
cr i ter i a. 

Thank  you  very  much  for  your  time  and  expertise. 


PART  I 


BACKGROUND 


A.  Position  of  the  Individual  (Job  Title,  Organization  or 
Office  Symbol) 


8.  Experience  in  your  present  job  as  well  as  total  Govern¬ 
ment  time. 


PART  II  —  SURVEY  QUESTIONS 

A.  Does  the  cost  of  the  replenishment  item  have  an  impact 
on  your  decision  to  purchase  acquisition  data? 

MUST  C  3  WANT  C  3 

0  1  2  3  4  5 


Low 

Importance/ 
No  Influence 

Desirable 

Important 

Vital 

COMMENTS: 

ft 

B.  Does  the  design  stability  of  the  item  based  on  the 
number  of  ECPs,  point  in  time  in  the  acquisition  process, 
the  maturity  of  technology  influence  your  decision?  What 
would  be  your  definition  of  design  stability? 

0 

MUST  C  3 

1  .  2 

WANT  C  3 

3  4 

5 

Low 

Importance/ 

Desirable 

Important 

Vital 

No  Influence 


COMMENTS: 


C.  Does  the  expected  life  of  the  item  in  the  inventory 
influence  your  decision  to  purchase  acquisition  data?  How 
long  or  short  must  the  time  be? 


MUST  C  3 

0  1  2 

Low  Desirable 

Importance/ 

No  Influence 

COMMENTS: 


WANT  C  3 
3  4 

Important 


5 

Vital 


D.  Does  the  purchase  cost  of  the  acquisition  data  have  an 
influence  on  your  decision?  What  dollar  ranges  are 
applicable? 

MUST  C  3 

0  1  2 

Low  Desirable 

Importance/ 

No  Influence 

COMMENTS: 


WANT  C 

T 


Important 


5 

Vital 


E.  What  are  your  organi zati onal /regul atory  requirements  to 
purchase  competitively  whenever  possible? 


0 


MUST  C  3 
1  2 


WANT  C 


5 


Low  Desirable  Important  Vital 

Importance/ 

No  Influence 


COMMENTS: 


F.  Does  the  possibility  that  the  original  supplier  may  go 
out  of  business  affect  your  decision?  On  what  do  you  base 
your  decision?  What  probability  o-f  the  vendor  going  out  o-f 
business  do  you  consider  acceptable? 


0 


MUST  C  3 
1  2 


WANT  C  3 
3  4 


Low  Desirable  Important  Vital 

Importance/ 

No  In-fluence 


COMMENTS: 


G.  Does  the  amount  o-f  -fixed  tooling  required  -for  the  item 
in-fluence  your  decision?  What  -factors  do  you  consider  in 
evaluating  the  amount  of  tooling  that  exists? 


0 

MUST  C  3 

1  2 

WANT  C  3 

3  4 

5 

Low 

Desirable 

Important 

Vital 

Importance/ 
No  Influence 

COMMENTS: 


H.  Does  the  anticipated  quantity  o-f  the  buy  influence  your 
decision?  What  quantity  level  is  considered  important? 


MUST  C  3 

WANT  C  3 

0 

1  2 

3  4 

5 

Low 

Desirable 

Important 

Vital 

Importance/ 
No  Influence 


COMMENTS: 


I.  Does  the  amount  of  critical  materials  required  -for  the 
item  influence  your  decision?  What  is  your  definition  of 
too  much  or  too  little  critical  materials? 


0 

MUST  C 

1 

3  WANT  C 

2  3 

3 

4 

5 

Low 

Importance/ 
No  Influence 

Desi rabl e 

Important 

Vital 

COMMENTS: 

J.  Does  cost  effectiveness  (time  and  personnel  requirements 
within  your  organization)  enter  into  your  decision  to  put — 
chase  acquisition  data?  What  are  these  consi derati ons? 

0 

MUST  C 

1 

3  WANT  C 

2  3 

3 

4 

5 

Low 

Desirable 

Important 

Vital 

Importance/ 
No  Influence 

COMMENTS: 


K.  Does  the  complexity  of  the  design  affect  your  decision? 
What  criteria  do  you  use  to  define  a  complex  item? 


0 

MUST  C  3 

1  2 

WANT  C  3 

3  4 

5 

Low 

Desirable 

Important 

Vital 

Importance/ 
No  Influence 


COMMENTS: 


L.  Are  there  other  criteria  that  you  consider  important  in 
your  decision  to  purchase  acquisition  data  which  have  not 
already  been  covered?  If  so,  what  are  they? 


MUST  C 


WANT  C 
3 


Low  Desirable 

Importance/ 

No  Influence 

COMMENTS: 


Important 


Vital 


PART  III  —  RANKING  THE  CRITERIA 

Please  rank,  the  following  criteria  in  order  of 
importance  (A  ranking  of  "l"  is  most  important).  There  may 
be  ties. 


Cri teria 


Rank 


Cost  of  the  replenishment  item 
Design  stability 
Expected  inventory  life 
Cost  of  the  acquisition  data 
Organizational /Regul atory  requirements 
Original  supplier's  business  solvency 
Amount  of  fixed  tooling 
Anticipated  quantity  of  the  buy 
Amount  of  critical  materials 

Time/personnel  reqrmts.  (Cost  effectiveness) 
Complexity  of  design 


Survey  #2 

Listed  below  are  eleven  criteria  used  by  data  personnel 
and  program  managers  when  deciding  whether  or  not  to 
purchase  acquisition  data.  These  criteria  are  listed  in 
order  of  importance  (#1  is  most  important;  #11  is  least 
important)  based  on  the  first  set  of  interviews  with 
acquisition  data  personnel. 

We  request  that  you  now  look  at  each  criterion 
individually  and  weight  each  from  one  to  ten,  with  a  weight 
of  one  meaning  that  the  criterion  is  relatively  unimportant 
in  deciding  whether  to  purchase  acquisition  data,  and  ten 
meaning  that  the  criterion  is  extremely  important.  00  NOT 
judge  one  criterion  against  another! 

IMPORTANT  ! • ! 

You  may  not  give  a  criterion  a  higher  weight  than  the 
weight  given  to  the  previous  criterion.  For  example,  if  you 
give  criterion  #1,  Design  Stability,  a  weight  of  9,  you  may 
give  criterion  #2,  Expected  Inventory  Life,  a  weight  of  9  or 
less.  Criterion  #1  need  not  be  weighted  as  a  ten.  Nor  need 
you  weight  criterion  #11  as  a  one.  A  sample  weighting  is 
provided  for  easier  understanding: 


NUM  CRITERIA _  WEI6HT 

1  Design  Stability  9 

2  Expected  Inventory  Life  7 

3  Complexity  of  Design  7 

4  Cost  of  the  Replenishment  Item  6 


10  Original  Supplier's  Business  Solvency  2 

11  Amount  of  Fixed  Tooling  2 


NUM  CRITERIA  WEIGHT 


1  Design  Stability 

2  Expected  Inventory  Life 

3  Complexity  of  Design 

4  Cost  of  the  Replenishment  Item 

5  Anticipated  Quantity  of  the  Buy 

6  Cost  of  the  Acquisition  Data 

7  Amount  of  Critical  Materials 

8  Organizational /Regulatory  Requirements 

9  Time/Personnel  Requirements  (Cost  Effectiveness) 

10  Original  Supplier's  Business  Solvency 

11  Amount  of  Fixed  Tooling 


Appendix  G:  Procurement  Method  Codes  and  Suffix  Codes 


Procurement 
Method  Code 


0 

1 


2 


3 


4 


Procurement 
Method 
Suffix  Code 


0 

A 


Explanation 


Not  established. 

Items  screened  and  found  to  be  already 
competi ti ve. 

Items  screened  and  determined  for  the 
first  time  to  be  suitable  for  competi¬ 
tive  procurement.  A  replenishment  item 
will  be  included  in  this  group  only  when 
the  identification  of  PMC  2  is  supported 
by  the  procurement  history  of  the  item. 
The  alternative  identification  is  PMC  1. 

Items  screened  and  found  to  be  procured 
directly  from  the  actual  manufacturer  or 
vendor,  including  a  prime  contractor  who 
is  the  actual  manufacturer. 

Items  screened  and  determined  for  the 
first  time  to  be  suitable  for  direct 
purchase  from  the  actual  manufacturer 
or  vendor  rather  than  the  original 
prime  contractor  for  the  end  items  which 
these  parts  support.  A  replenishment 
item  will  be  included  in  this  group  only 
when  the  identification  as  PMC  4  is 
supported  by  the  procurement  history  of 
the  item.  The  alternative  identifica¬ 
tion  is  PMC  3. 

Items  screened  and  determined  not 
suitable  for  competitive  procurement  or 
direct  purchase  and  which,  therefore, 
continue  to  be  procured  from  a  prime 
contractor  who  is  not  the  actual 
manuf acturer . 


_ Explanation _ 

Not  established. 

Government’s  rights  in  data 
questionable. 


B 


Source  control 


Procurement 

Method 

Suffix  Code  _ Explanation 


C 


Procurement  -from  approved  source. 


0 

E 

F 

G 

H 

J 

K 

L 

M 

N 

P 

Q 

R 

S 

T 


The  data  not  available. 

Status  can  be  improved. 

This  item  is  phased  provisioning. 

Data  is  technically  suitable  and  legally 
clear. 

Inadequate  data. 

Restricted  to  the  prime  contractor. 
Produced  -from  class  1A  castings. 

Low  dollar  value  o-f  procurement. 

Master  or  coordinated  tooling. 

Requires  special  test. 

Rights  to  use  data  legally  unavailable. 

Requires  exceptional  unique  manufactur¬ 
ing  processes. 

Rights  to  use  data  restricted. 

Security  classification. 

Qualified  Products  List  (QPL) . 

This  item  is  uneconomical  to  compete. 

High  reliability  part. 

Parts  may  be  procured  by  the  method 
indicated  by  the  PMC  if  military  or 
adopted  industry  specifications  are 
substituted  for  the  contractor’s  data 
which  are  subject  to  limited  use  rights. 


Design  unstable. 

Necessary  to  ensure  standardization  and 
inter changeabi 1 ity . 


Appendix  Hs  Selected  Experts*  Comments  on  the 

Decision  Cri teria 


1 .  Design  Stabi 1 i ty: 

-  A  stable  design  is  achieved  whenever  the  item 
achieves  all  of  the  requirements  specified  in  the 
Part  I  5c  1 1  specifications. 

-  A  stable  design  can  never  really  be  achieved.  It  is 
only  stable  in  relative  terms. 

-  A  stable  design  occurs  when  the  item  is  baselined 
for  production. 

2.  Expected  Inventory  Life; 

-  Anything  expected  to  be  replaced  in  4  years  or  less 
should  be  supported  by  the  contractor. 

-  We  us  a  rough  estimate  of  7  years  in  the  inventory 
as  a  guide. 

-  To  purchase  data  the  item  should  be  in  the  inventory 
at  least  3  years  after  the  production  line  stops. 

-  Approximately  5  years. 

3.  Complexity  of  Design: 

-  If  the  Air  Force  does  not  have  the  capability  to 
maintain  the  item  because  of  complexity  a  data 
package  is  useless. 

-  You  cannot  define  or  estimate  "complex". 

-  With  highly  complex  items  it  may  be  impossible  to 
use  a  data  package  to  procure  from  another  source 
because  the  complexity  of  the  data  keeps  the 
government  from  validating  its  completeness. 

-  We  use  a  "guesstimate"  as  to  item  complexity. 

4.  Cost  of  the  Replenishment  Item: 

-  When  purchasing  data  cost  is  always  a  concern. 

-  From  a  configuration  management  viewpoint,  the  cost 
of  the  item  is  not  important,  if  AFLC  wants  the  data 
we  put  it  on  the  contract. 


5.  Anticipated  Quantity  of  the  Buy: 

-  Small  quantities  do  not  justify  acquisition  data. 

-  There  is  a  trade-off  between  cost  and  the  number  and 
use  of  the  item. 


-  The  purchase  of  the  appropriate  data  is  independent 
of  the  number  of  items. 

-  Especially  important  with  large,  complex,  costly 
items  the  Air  Force  will  need  the  same  data  to 
support  one  item  as  1000  of  the  items. 

6.  Cost  of  the  Acquisition  Data: 

-  An  excessively  high  cost  may  preclude  purchase,  but 
"excessively  high"  is  a  function  of  the  type  data. 

-  Anything  over  10%  of  the  cost  of  the  item  is  too 
much  to  pay  for  the  data. 

-  The  cost  of  the  data  should  not  exceed  the  cost  of 
redesigning  the  item. 

-  As  a  rule,  data  cost  should  not  exceed  40%  of  the 
overall  contract  price. 

7.  Amount  of  Critical  Materials: 

-  No  comments  were  received  concerning  this  criterion. 

8.  Organisational /Regulatory  Requirements: 

-  Competition  is  in,  sole  source  is  out. 

-  Competition  is  required  to  the  maximum  extent. 

9.  T i me/Per sonnel  Requirements  (Cost  Effectiveness) : 

-  Unfortunately,  a  lack  of  people  sometimes  forces  us 
to  not  purchase  data. 

-  Not  a  factor,  if  data  is  needed  it  is  bought. 

-  If  data  is  essential  it  is  purchased. 

10.  Original  Suppl ier ’ s  Business  Solvency: 


Fixture  peculiarity  and  adaptability  are  major 
considerations. 

Not  a  relevant  -factor. 

A  manufacturing  concern,  not  a  -factor  in  data 
procurement . 
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